diff --git a/Readme.md b/Readme.md index a4319d1..85c2dfa 100644 --- a/Readme.md +++ b/Readme.md @@ -27,6 +27,12 @@ Version 1 will have C and a subset of C++ features available to it. I will generate with this library a C99 or 11 variant when Version 1 is complete. A single-header version will also be generated. +The size target of this library is to stay under 5000 sloc (data & interface code). +With the dependency code being under 10000 sloc. (Containers, Memory, String handling, Language bloat) + +Any dependencies from the zpl library will be exposed manually with using declarations into global scope. +They will be removed when the library is feature complete for version 1 (zero dependencies milestone). + ## Usage A metaprogram is built to generate files before the main program is built. We'll term runtime for this program as `gen_time`. The metaprogram's core implementation are within `gen.hpp` and `gen.cpp` in the project directory. @@ -205,46 +211,53 @@ If in your use case, decide to have exclusive separation or partial separation o ### *WHAT IS NOT PROVIDED* -* Macro or template generation : This library is to avoid those, adding support for them adds unnecessary complexity. - If you desire define them outside the gen_time scopes. -* Expression validation : Execution expressions are defined using the untyped string API. - There is no parse API for validating expressions (possibly will add in the future) -* Modern C++ (STL library) features -* Modern C++ RTTI : This is kinda covered with the last point, but just wanted to emphasize. +* Macro or template generation : This library is to avoid those, adding support for them adds unnecessary complexity. +* Vendor provided dynamic dispatch (virtuals) : Roll your own, this library might roll its own vtable/interface generation helpers in the future. +* RTTI : This is kinda covered with the last point, but just wanted to emphasize. +* Exceptions : Most fo the +* Execution statment validation : Execution expressions are defined using the untyped string API. -Exceptions brought in from "Modern C++": +Keywords in from "Modern C++": -* consteval -* constinit -* explicit -* export -* noexcept -* import -* final -* module -* override -* && -* virtual +* constexpr : Great to store compile-time constants, (easier to garanteed when emitted from gentime) +* consteval : Technically fine so long as templates are not used. Need to make sure to execute in moderation. +* constinit : Better than constexpr at doing its job, however, its only c++ 20. +* export : Useful if c++ modules ever come around to actually being usable. +* import : ^^ +* module : ^^ + +These features are in as they either are not horrible when used conservatively or are a performance benefit (modules). + +When it comes to excution statements: +There is no parse API for validating excution statements (possibly will add in the future, but very limited in what it can do). +This reason there isn't one: thats where the can of worms open for parsing validation. +For most metaprogramming (espcially for c/c++), expression validation is not necessary, it can be done by the compiler for the runtime program. +Most of the time, the critical complex metaprogramming conundrums are actaully producing the frame of abstractions around the expressions. +Thus its not very much a priority to add such a level of complexity to the library when there isn't a high reward or need for it. + +To further this point, lets say you do have an error with an execution statment. It will either be caught by the c++ compiler when compiling the target program, or at runtime for the program. + +* If its not caught by the compiler, the only downside is the error appers on the generated function. Those with knowledge of how that definition was generated know where to find the code that inlined that expression in that file for that definition. +* If its caught at runtime. The expression will be shown in a stack trace if debug symbols are enabled in the generated function body. Yet again those with knowledge of how that definition was generated know where to find the code that inlined that expression. + +In both these cases will get objectively better debug information than you would normally get on most c++ compilers with complex macros or templates. + +### The Data & Interface As mentioned in [Usage](#Usage), the user is provided Code objects by calling the constructor functions to generate them or find existing matches. -The AST is managed by the library, however the user may specificy memory configuration. - -Notes: - -* The allocator definitions used are exposed to the user incase they want to dictate memory usage* -* ASTs are wrapped for the user in a Code struct which essentially a warpper for a AST* type. -* Both AST and Code have member symbols but their data layout is enforced to be POD types. +The AST is managed by the library and provided the user via its interface prodedures. +However, the user may specificy memory configuration. Data layout of AST struct: ```cpp AST* Parent; -string_const Name; -string_const Comment; +CachedString Name; +CachedString Comment; union { array(AST*) Entries; - string_const Content; + CachedString Content; }; CodeT Type; OperatorT Op; @@ -252,12 +265,25 @@ bool Readonly; u8 _64_Align[23]; ``` -*`CodeT` is a typedef for `ECode::Type` which has an underlying type of u32* -*`OperatorT` is a typedef for `EOperator::Type` which has an underlying type of u32.* +*`CodeT` is a typedef for `ECode::Type` which has an underlying type of `u32`* +*`OperatorT` is a typedef for `EOperator::Type` which has an underlying type of `u32`* ASTs can be set to readonly by calling Code's lock() member function. Adding comments is always available even if the AST is set to readonly. +Data Notes: + +* The allocator definitions used are exposed to the user incase they want to dictate memory usage + * You'll find the memory handling in `init`, `gen_string_allocator`, `get_cached_string`, `make_code`, and `make_code_entries`. +* ASTs are wrapped for the user in a Code struct which essentially a warpper for a AST* type. +* Both AST and Code have member symbols but their data layout is enforced to be POD types. +* This library treats memory failures as fatal. +* The default setup assumes large definition sets may be provided to bodies so AST::Entires are dynamic arrays. + * They're allocated to arenas currently and are pretty wasteful if they go over their reserve size (its never recycled). + * Most likely will need to implement a dynamic-sized bucket allocation strategy for the entry arrays if memory is getting stressed. + * Otherwise if you are using fixed size entries and your definitions are under 128~512 entries for the body, you may be better of with a fixed-sized array. +* Strings are stored in their own set of arenas. AST constructors use cached strings for names, and content. + ## There are four sets of interfaces for Code AST generation the library provides * Upfront @@ -439,14 +465,14 @@ There are three provided interfaces: Editor and Scanner are disabled by default, use `GEN_FEATURE_EDITOR` and `GEN_FEATURE_SCANNER` to enable them. -### Builder is a similar object to the jai language's string_builder. +### Builder is a similar object to the jai language's string_builder * The purpose of it is to generate a file. * A file is specified and opened for writting using the open( file_path) ) function. * The code is provided via print( code ) function will be seralized to its buffer. * When all seralization is finished, use the write() command to write the buffer to the file. -### Editor is for editing a series of files based on a set of requests provided to it. +### Editor is for editing a series of files based on a set of requests provided to it * The purpose is to overrite a specific file, it places its contents in a buffer to scan. * Requests are populated using the following interface: @@ -468,7 +494,7 @@ Additionally if `GEN_FEATURE_EDITOR_REFACTOR` is defined, refactor( file_path, s Refactor is based of the refactor library and uses its interface. It will on call add a request to the queue to run the refactor script on the file. -### Scanner allows the user to generate Code ASTs by reading files. +### Scanner allows the user to generate Code ASTs by reading files * The purpose is to grab definitions to generate metadata or generate new code from these definitions. * Requests are populated using the add( SymbolInfo, Policy ) function. The symbol info is the same as the one used for the editor. So is the case with Policy. @@ -487,7 +513,7 @@ This is intended for when you have requests that are for multiple files. Request queue in both Editor and Scanner are cleared once process_requests completes. -## On multi-threading: +## On multi-threading Its intended eventually for this library to support multi-threading at some point, however for now it does not. @@ -543,12 +569,20 @@ However, if: Then this might help you boostrap a toolset todo so. -# TODO: +# TODO -* Need problably a better name, I found a few repos with this same one... -* Actually get to version 1. +* May be in need of a better name, I found a few repos with this same one... * Make a test suite made up of collections based of the ZPL library templated colllection macros and the memory module. +* Remove full ZPL dependency, move into Bloat header/source only what is used. * Generate a single-header library. * Generate a C-supported single-header library. -* Remove full ZPL dependency, move into Bloat header/source only what is used. -* This library has heavy string allocations, most likely will make a string flyweight for it. +* Actually get to version 1. +* Review if the upfront or incremental constructors are actually a net benefit vs just using the parse constructors. + * They exist as a artifact of learning what was possible or not possible with staged metaprogramming in C++ (the parse interface was the last to get fleshed out) + * Most likely at least Incremental could possibly be removed in favor of just using the parse constructors. + * Possible merits are ergonomics for very dynamic generation or performance reasons. + * They'll most likely stay until its evident that they are not necessary. +* Review memory handling for the AST, specifically relating to: + * Giving type asts a dedicated memory arenas. + * Giving specifier definitions a dedicated memory arenas and hashtable lookup. + * Possibly adding a dedicated block allocator for the dynamic arrays of AST::Entires. diff --git a/gencpp.code-workspace b/gencpp.code-workspace deleted file mode 100644 index 7f95ff0..0000000 --- a/gencpp.code-workspace +++ /dev/null @@ -1,20 +0,0 @@ -{ - "folders": [ - { - "path": "." - } - ], - "settings": { - "files.associations": { - "*.rmd": "markdown", - "array": "cpp", - "compare": "cpp", - "type_traits": "cpp", - "utility": "cpp", - "xtr1common": "cpp", - "xutility": "cpp", - "initializer_list": "cpp", - "table.h": "c" - } - } -} \ No newline at end of file diff --git a/project/Bloat.hpp b/project/Bloat.hpp index 6fc5c22..6d7b731 100644 --- a/project/Bloat.hpp +++ b/project/Bloat.hpp @@ -53,11 +53,11 @@ using zpl::FileInfo; using zpl::FileError; using zpl::Pool; using zpl::String; -// using zpl::StringHeader; using zpl::EFileMode_WRITE; using zpl::EFileError_NONE; +using zpl::alloc; using zpl::arena_allocator; using zpl::arena_init_from_memory; using zpl::arena_free; @@ -65,6 +65,8 @@ using zpl::char_is_alpha; using zpl::char_is_space; using zpl::crc32; using zpl::memset; +using zpl::pool_allocator; +using zpl::pool_init; using zpl::pool_free; using zpl::printf_va; using zpl::printf_err_va; @@ -75,7 +77,7 @@ using zpl::string_append_length; using zpl::string_length; using zpl::string_make; using zpl::strnlen; -// using zpl:: +using zpl::exit; #if __clang__ diff --git a/project/gen.cpp b/project/gen.cpp index 7e7c3d1..5adb665 100644 --- a/project/gen.cpp +++ b/project/gen.cpp @@ -1,7 +1,7 @@ #include "Bloat.hpp" #include "gen.hpp" -#ifdef gen_time +#ifdef gentime namespace gen { ZPL_TABLE_DEFINE( StringTable, str_tbl_, String ); @@ -9,85 +9,22 @@ namespace gen namespace StaticData { - #ifndef GEN_CODE_USE_SOA - static Array(Pool) CodePools; - static Array(Pool) CodeEntriesPools; - - #else - using DataUnion = union - { - array(AST*) Entries; - string Content; - }; - - namespace CodePool - { - array(CodeT) Type; - array(bool) Readonly; - array(AST*) Parent; - array(string) Name; - array(string) Comment; - array(DataUnion) Data; - - forceinline - CodeT& type( sw index ) - { - return Type[index]; - } - - forceinline - bool& readonly( sw index ) - { - return Readonly[index]; - } - - forceinline - AST*& parent( sw index ) - { - return Parent[index]; - } - - forceinline - string& name( sw index ) - { - return Name[index]; - } - - forceinline - string& comment( sw index ) - { - return Comment[index]; - } - - forceinline - DataUnion& data( sw index ) - { - return Data[index]; - } - } - #endif - - static Array(Arena) StringArenas = nullptr; + static Array(Pool) CodePools = nullptr; + static Array(Arena) CodeEntriesArenas = nullptr; + static Array(Arena) StringArenas = nullptr; static StringTable StringMap; static TypeTable TypeMap; - - static sw InitSize_CodePool = megabytes(64); - static sw InitSize_CodeEntriesPool = megabytes(8); - static sw InitSize_StringTable = megabytes(4); - static sw InitSize_TypeTable = megabytes(4); - - static sw SizePer_StringArena = megabytes(32); - - static AllocatorInfo Allocator_CodePool = heap(); - static AllocatorInfo Allocator_CodeEntriesPool = heap(); - static AllocatorInfo Allocator_StringArena = heap(); - static AllocatorInfo Allocator_StringTable = heap(); - static AllocatorInfo Allocator_TypeTable = heap(); + static AllocatorInfo Allocator_DataArrays = heap(); + static AllocatorInfo Allocator_CodePool = heap(); + static AllocatorInfo Allocator_CodeEntriesArena = heap(); + static AllocatorInfo Allocator_StringArena = heap(); + static AllocatorInfo Allocator_StringTable = heap(); + static AllocatorInfo Allocator_TypeTable = heap(); } -#pragma region CONSTANTS +#pragma region Constants # ifdef GEN_DEFINE_LIBRARY_CODE_CONSTANTS Code type_ns(void); @@ -112,20 +49,50 @@ namespace gen Code type_ns(f64); # endif + Code access_public; + Code access_protected; + Code access_private; + Code spec_constexpr; + Code spec_const; Code spec_inline; -#pragma endregion CONSTANTS + Code sepc_ptr; + Code spec_ref; +#pragma endregion Constants # pragma region AST + const Code Code::Invalid; + bool AST::add( AST* other ) { switch ( Type ) { using namespace ECode; + case Invalid: + break; + case Untyped: break; + case Access_Public: + break; + + case Access_Protected: + break; + + case Access_Private: + break; + + case Class: + break; + + case Class_FwdDecl: + break; + + case Class_Body: + break; + case Enum: break; @@ -141,7 +108,7 @@ namespace gen case Enum_Class_FwdDecl: break; - case Global_Body: + case Friend: break; case Function: @@ -153,12 +120,21 @@ namespace gen case Function_FwdDecl: break; + case Global_Body: + break; + case Namespace: break; case Namespace_Body: break; + case Operator: + break; + + case Operator_FwdDecl: + break; + case Parameters: break; @@ -196,10 +172,31 @@ namespace gen { using namespace ECode; + case Invalid: + break; + case Untyped: break; - case Global_Body: + case Access_Public: + break; + + case Access_Protected: + break; + + case Access_Private: + break; + + case Enum: + break; + + case Enum_FwdDecl: + break; + + case Enum_Body: + break; + + case Friend: break; case Function: @@ -211,12 +208,21 @@ namespace gen case Function_FwdDecl: break; + case Global_Body: + break; + case Namespace: break; case Namespace_Body: break; + case Operator: + break; + + case Operator_FwdDecl: + break; + case Parameters: break; @@ -242,7 +248,6 @@ namespace gen break; } - return true; } @@ -262,6 +267,10 @@ namespace gen switch ( Type ) { + case Invalid: + + break; + case Untyped: case Access_Public: case Access_Protected: @@ -272,7 +281,7 @@ namespace gen case Specifiers: // Can just be the same, as its a cached string. result->Content = Content; - break; + return; // The main purpose of this is to make sure entires in the AST are unique, // So that we can assign the new parent without corrupting the existing AST. @@ -294,7 +303,7 @@ namespace gen case Typedef: case Typename: case Using: - array_init( result->Entries, StaticData::Allocator_CodePool ); + result->Entries = make_code_entries(); s32 index = 0; s32 left = array_count( result->Entries ); while ( left -- ) @@ -306,7 +315,7 @@ namespace gen result->Entries[index]->Parent = this; index++; } - break; + return; } } @@ -498,32 +507,73 @@ namespace gen return result; } - - const Code Code::Invalid; # pragma endregion AST #pragma region Gen Interface void init() { - array_init( StaticData::CodePools, StaticData::Allocator_CodePool ); - array_init( StaticData::StringArenas, heap() ); + using namespace StaticData; - Arena string_arena; - arena_init_from_allocator( & string_arena, StaticData::Allocator_StringArena, StaticData::SizePer_StringArena ); + // Setup the arrays + { + if (! array_init_reserve( CodePools, Allocator_DataArrays, InitSize_DataArrays ) ) + fatal( "gen::init: Failed to initialize the CodePools array" ); - str_tbl_init( & StaticData::StringMap, StaticData::Allocator_StringTable ); - type_tbl_init( & StaticData::TypeMap, StaticData::Allocator_TypeTable ); + if ( ! array_init_reserve( CodeEntriesArenas, Allocator_DataArrays, InitSize_DataArrays ) ) + fatal( "gen::init: Failed to initialize the CodeEntriesPools array" ); + + if ( ! array_init_reserve( StringArenas, Allocator_DataArrays, InitSize_DataArrays ) ) + fatal( "gen::init: Failed to initialize the StringArenas array" ); + } + + // Setup the code pool and code entries arena. + { + Pool code_pool; + pool_init( & code_pool, Allocator_CodePool, CodePool_NumBlocks, sizeof(AST) ); + + if ( code_pool.physical_start == nullptr ) + fatal( "gen::init: Failed to initialize the code pool" ); + + array_append( CodePools, code_pool ); + + Arena code_entires_arena; + arena_init_from_allocator( & code_entires_arena, Allocator_CodeEntriesArena, SizePer_CodeEntriresArena ); + + if ( code_entires_arena.physical_start == nullptr ) + fatal( "gen::init: Failed to initialize the code entries arena" ); + + array_append( CodeEntriesArenas, code_entires_arena ); + + Arena string_arena; + arena_init_from_allocator( & string_arena, Allocator_StringArena, SizePer_StringArena ); + + if ( string_arena.physical_start == nullptr ) + fatal( "gen::init: Failed to initialize the string arena" ); + + array_append( StringArenas, string_arena ); + } + + // Setup the hash tables + { + str_tbl_init ( & StringMap, Allocator_StringTable ); + if ( StringMap.entries == nullptr ) + fatal( "gen::init: Failed to initialize the StringMap"); + + type_tbl_init( & TypeMap, Allocator_TypeTable ); + if ( TypeMap.entries == nullptr ) + fatal( "gen::init: Failed to initialize the TypeMap" ); + } Code& InvalidCode_write = ccast( Code, Code::Invalid ); InvalidCode_write = make_code(); - #ifdef GEN_DEFINE_LIBRARY_CODE_CONSTANTS +# ifdef GEN_DEFINE_LIBRARY_CODE_CONSTANTS Code& t_bool_write = ccast( Code, t_void ); t_bool_write = def_type( txt(void) ); - # define def_constant_code_type( Type_ ) \ +# define def_constant_code_type( Type_ ) \ Code& \ t_##Type_ = def_type( txt(Type_) ); \ t_##Type_.lock() @@ -547,52 +597,57 @@ namespace gen def_constant_code_type( f32 ); def_constant_code_type( f64 ); - # undef def_constant_code_type - #endif +# undef def_constant_code_type +# endif Code& spec_constexpr_write = ccast( Code, spec_constexpr ); spec_constexpr_write = def_specifiers( 1, ESpecifier::Constexpr ); - # define def_constant_spec( Type_, ... ) \ +# define def_constant_spec( Type_, ... ) \ Code& \ spec_##Type_ = def_specifiers( macro_num_args(__VA_ARGS__), __VA_ARGS__); \ spec_##Type_.lock(); def_constant_spec( const, ESpecifier::Const ); def_constant_spec( inline, ESpecifier::Inline ); - # undef def_constant_spec +# undef def_constant_spec } void clear_code_pools() { + using namespace StaticData; + // Clear the code pools { s32 index = 0; - s32 left = 0; + s32 left = 0; while (( left-- )) { - Pool* code_pool = & StaticData::CodePools[index]; + Pool* code_pool = & CodePools[index]; pool_free( code_pool ); } - array_clear( StaticData::CodePools ); + array_clear( CodePools ); } // Clear the code entries pools { s32 index = 0; - s32 left = 0; + s32 left = 0; while (( left-- )) { - Pool* code_entries_pool = & StaticData::CodeEntriesPools[index]; - pool_free( code_entries_pool ); + Arena* code_entries_arena = & CodeEntriesArenas[index]; + arena_free( code_entries_arena ); } - array_clear( StaticData::CodeEntriesPools ); + array_clear( CodeEntriesArenas ); } + + type_tbl_clear( & StaticData::TypeMap ); } + inline AllocatorInfo get_string_allocator( s32 str_length ) { using namespace StaticData; @@ -602,7 +657,8 @@ namespace gen Arena new_arena; arena_init_from_allocator( & new_arena, Allocator_StringArena, SizePer_StringArena ); - array_append( StringArenas, new_arena ); + if ( array_append( StringArenas, new_arena ) ) + fatal( "gen::get_string_allocator: Failed to allocate a new string arena" ); return arena_allocator( StringArenas ); } @@ -611,18 +667,14 @@ namespace gen } // Will either make or retrive a code string. - string_const cached_string( char const* cstr, s32 length ) + StringCached get_cached_string( char const* cstr, s32 length ) { - s32 hash_length = length > kilobytes(1) ? kilobytes(1) : length; - - u32 key = crc32( cstr, hash_length ); - - String* result = str_tbl_get( & StaticData::StringMap, key ); + s32 hash_length = length > kilobytes(1) ? kilobytes(1) : length; + u32 key = crc32( cstr, hash_length ); + String* result = str_tbl_get( & StaticData::StringMap, key ); if ( result ) - { return * result; - } * result = string_make( get_string_allocator( length ), cstr ); @@ -638,95 +690,172 @@ namespace gen { using namespace StaticData; -# ifndef GEN_CODE_USE_SOA ct CodePOD Invalid = { nullptr, nullptr, nullptr, nullptr, ECode::Invalid, EOperator::Invalid, false, {0} }; + AllocatorInfo allocator = { nullptr, nullptr }; + s32 left = array_count( CodePools ); + do + { + if ( CodePools[left].free_list != nullptr ) + allocator = zpl::pool_allocator( & CodePools[left] ); + } + while ( left--, left ); -# else + if ( allocator.data == nullptr ) + { + Pool code_pool; + pool_init( & code_pool, Allocator_CodePool, CodePool_NumBlocks, sizeof(AST) ); - array_append( CodePool::Type, ECode::Invalid ); - array_append( CodePool::Readonly, false ); - array_append( CodePool::Name, nullptr ); - array_append( CodePool::Comment, nullptr ); - array_append( CodePool::Data, { nullptr } ); + if ( code_pool.physical_start == nullptr ) + fatal( "gen::make_code: Failed to allocate a new code pool - CodePool allcoator returned nullptr." ); - Code code { array_count( CodePool::Type) - 1 }; + if ( ! array_append( CodePools, code_pool ) ) + fatal( "gen::make_code: Failed to allocate a new code pool - CodePools failed to append new pool." ); - return code; -# endif + allocator = pool_allocator( CodePools ); + } + + Code result { rcast( AST*, alloc( allocator, sizeof(AST) )) }; + + * result = pcast( AST, Invalid); + + return result; } Array(AST*) make_code_entries() { + using namespace StaticData; + AllocatorInfo allocator = { nullptr, nullptr }; + + s32 left = array_count( CodeEntriesArenas ); + do + { + if ( arena_size_remaining(CodeEntriesArenas, ZPL_DEFAULT_MEMORY_ALIGNMENT) > ZPL_ARRAY_GROW_FORMULA(0) ) + allocator = arena_allocator( & CodeEntriesArenas[left] ); + } + while( left--, left ); + + if ( allocator.data == nullptr ) + { + Arena arena; + arena_init_from_allocator( & arena, Allocator_CodeEntriesArena, SizePer_CodeEntriresArena ); + + if ( arena.physical_start == nullptr ) + fatal( "gen::make_code: Failed to allocate a new code entries arena - CodeEntriesArena allcoator returned nullptr." ); + + allocator = arena_allocator( CodeEntriesArenas ); + } + + Array(AST*) entry_array; + array_init( entry_array, allocator ); + + return entry_array; } + forceinline bool operator_member_symbol_check( Code entry ) { + using namespace ECode; + // Is assumed by the time this is called, entry has been valided to not be null or invalid. + switch ( entry->Type ) + { + case Access_Public: + case Access_Protected: + case Access_Private: + case Class_Body: + case Class_FwdDecl: + case Enum_FwdDecl: + case Enum_Body: + case Enum_Class_FwdDecl: + case Friend: + case Function: + case Function_FwdDecl: + case Global_Body: + case Namespace: + case Namespace_Body: + case Operator: + case Operator_FwdDecl: + case Parameters: + case Specifiers: + case Struct_FwdDecl: + case Struct_Body: + case Typename: + return false; + } + + return true; } - void set_allocator_code_pool( AllocatorInfo pool_allocator ) + void set_allocator_data_arrays( AllocatorInfo allocator ) { - StaticData::Allocator_CodePool = pool_allocator; + StaticData::Allocator_DataArrays = allocator; } - void set_allocator_string_arena( AllocatorInfo string_allocator ) + void set_allocator_code_pool( AllocatorInfo allocator ) { - StaticData::Allocator_StringArena = string_allocator; + StaticData::Allocator_CodePool = allocator; } - void set_allocator_string_table( AllocatorInfo string_allocator ) + void set_allocator_code_entries_arena( AllocatorInfo allocator ) { - StaticData::Allocator_StringArena = string_allocator; + StaticData::Allocator_CodeEntriesArena = allocator; } - void set_init_reserve_code_pool( sw size ) + void set_allocator_string_arena( AllocatorInfo allocator ) { - StaticData::InitSize_CodePool = size; + StaticData::Allocator_StringArena = allocator; } - void set_init_reserve_string_table( sw size ) + void set_allocator_string_table( AllocatorInfo allocator ) { - StaticData::InitSize_StringTable = size; - } - - void set_init_reserve_type_table( sw size ) - { - StaticData::InitSize_TypeTable = size; - } - - void set_allocator_type_table( AllocatorInfo type_reg_allocator ) - { - StaticData::Allocator_TypeTable = type_reg_allocator; - } - - void set_size_string_arena( sw size ) - { - StaticData::SizePer_StringArena = size; + StaticData::Allocator_StringArena = allocator; } # pragma region Helper Functions - // This snippet is required in nearly all the functions. -# define name_check( Context_, Length_, Name_ ) \ - do \ - { \ - if ( Length_ <= 0 ) \ - { \ - log_failure( "gen::%s: Invalid name length provided - %d", #Context_, length ); \ - return Code::Invalid; \ - } \ - \ - if ( Name_ == nullptr ) \ - { \ - log_failure( "gen::%s: name is null", #Context_); \ - return Code::Invalid; \ - } \ - } \ - while (0) + // This snippet is used in nearly all the functions. +# define name_check( Context_, Length_, Name_ ) \ + { \ + if ( Length_ <= 0 ) \ + { \ + log_failure( "gen::%s: Invalid name length provided - %d", txt(Context_), length ); \ + return Code::Invalid; \ + } \ + \ + if ( Name_ == nullptr ) \ + { \ + log_failure( "gen::%s: name is null", txt(Context_) ); \ + return Code::Invalid; \ + } \ + } +# define null_check( Context_, Code_ ) \ + if ( ! Code_ ) \ + { \ + log_failure( "gen::%s: %s provided is null!", txt(Context_), txt(Code_) ); \ + return Code::Invalid; \ + } +# define null_or_invalid_check( Context_, Code_ ) \ + { \ + if ( ! Code_ ) \ + { \ + log_failure( "gen::%s: %s provided is null!", txt(Context_) ); \ + return Code::Invalid; \ + } \ + \ + if ( Code_->is_invalid() ) \ + { \ + log_failure("gen::%s: %s provided is invalid!", txt(Context_), txt(Code_) ); \ + return Code::Invalid; \ + } \ + } + +# define not_implemented( Context_ ) \ + log_failure( "gen::%s: This function is not implemented" ); \ + return Code::Invalid; # pragma endregion Helper Functions # pragma region Upfront Constructors @@ -750,7 +879,7 @@ namespace gen Code result = make_code(); - result->Name = cached_string( name, length ); + result->Name = get_cached_string( name, length ); if ( body ) { @@ -798,7 +927,7 @@ namespace gen Code result = make_code(); - result->Name = cached_string( name, length ); + result->Name = get_cached_string( name, length ); if ( body ) { @@ -834,22 +963,31 @@ namespace gen return result; } - Code def_friend( Code symbol ) + Code def_execution( Code untyped_code ) + { + null_check( def_execution, untyped_code ); + + if ( untyped_code->Type != ECode::Untyped ) + { + log_failure( "gen::def_execution: untyped_code is not of untyped type - %s", untyped_code->debug_str() ); + return Code::Invalid; + } + + Code + result = make_code(); + result->Type = ECode::Execution; + + result.lock(); + return result; + } + + Code def_friend( Code declaration ) { using namespace ECode; - if ( ! symbol ) - { - log_failure( "gen::def_friend: symbol provided is null!" ); - } + null_check( def_friend, declaration ); - if ( symbol == Code::Invalid ) - { - log_failure("gen::def_friend: symbol provided is invalid!" ); - return; - } - - switch ( symbol->Type ) + switch ( declaration->Type ) { case Class_FwdDecl: case Function_FwdDecl: @@ -858,11 +996,20 @@ namespace gen break; default: - log_failure("gen::def_friend: symbol cannot be used with friend, must be a forward declare - %s", symbol->debug_str()); - return; + // Technically friends can have a function body. I will not support it so if you want that add it yourself. - Ed. + log_failure("gen::def_friend: declaration cannot be used with friend, must be a forward declare - %s", declaration->debug_str()); + return Code::Invalid; } + Code + result = make_code(); + result->Type = Friend; + result->Entries = make_code_entries(); + result->add_entry( declaration ); + + result.lock(); + return result; } Code def_function( s32 length, char const* name @@ -895,10 +1042,9 @@ namespace gen } Code - result = make_code(); - result->Name = cached_string( name, length ); - - array_init( result->Entries, StaticData::Allocator_CodePool ); + result = make_code(); + result->Name = get_cached_string( name, length ); + result->Entries = make_code_entries(); if ( body ) { @@ -947,13 +1093,13 @@ namespace gen using namespace ECode; name_check( def_namespace, length, name ); + null_check( def_namespace, body ); Code - result = make_code(); - result->Type = Namespace; - result->Name = cached_string( name, length ); - - array_init( result->Entries, g_allocator ); + result = make_code(); + result->Type = Namespace; + result->Name = get_cached_string( name, length ); + result->Entries = make_code_entries(); if ( body->Type != Namespace_Body || body->Type != Untyped ) { @@ -988,12 +1134,46 @@ namespace gen break; } - Code result = make_code(); + Code + result = make_code(); + result->Type = ECode::Operator; + + if ( body ) + { + + } + else + { + + } + + result.lock(); + return result; } Code def_param( Code type, s32 length, char const* name ) { + using namespace ECode; + name_check( def_param, length, name ); + null_check( def_param, type ); + + if ( type->Type != Typename ) + { + log_failure( "gen::def_param: type is not a typename - %s", type->debug_str() ); + return Code::Invalid; + } + + Code + result = make_code(); + result->Type = Parameters; + result->Name = get_cached_string( name, length ); + result->Entries = make_code_entries(); + + result->add_entry( type ); + + result.lock(); + return result; } Code def_struct( u32 length, char const* name, Code body, Code parent, Code specifiers ) @@ -1022,12 +1202,18 @@ namespace gen Code result = make_code(); - result->Type = Struct; - result->Name = cached_string( name, length ); + result->Name = get_cached_string( name, length ); result->Entries = make_code_entries(); if ( body ) + { + result->Type = Struct; result->add_entry( body ); + } + else + { + result->Type = Struct_FwdDecl; + } if ( parent ) result->add_entry( parent ); @@ -1041,6 +1227,7 @@ namespace gen Code def_variable( Code type, u32 length, char const* name, Code value, Code specifiers ) { name_check( def_variable, length, name ); + null_check( def_variable, type ); if ( specifiers && specifiers->Type != ECode::Specifiers ) { @@ -1062,7 +1249,7 @@ namespace gen Code result = make_code(); - result->Name = cached_string( name, length ); + result->Name = get_cached_string( name, length ); result->Type = ECode::Variable; result->Entries = make_code_entries(); @@ -1081,26 +1268,51 @@ namespace gen { name_check( def_type, length, name ); + if ( specifiers && specifiers->Type != ECode::Specifiers ) + { + log_failure( "gen::def_type: specifiers is not of specifiers type - %s", specifiers->debug_str() ); + return Code::Invalid; + } + Code result = make_code(); - result->Name = cached_string( name, length ); + result->Name = get_cached_string( name, length ); result->Type = ECode::Typename; + if ( specifiers ) + { + result->Entries = make_code_entries(); + result->add_entry( specifiers ); + } + + result.lock(); return result; } Code def_using( u32 length, char const* name, Code type, UsingT specifier ) { name_check( def_using, length, name ); + null_check( def_using, type ); Code result = make_code(); - result->Name = cached_string( name, length ); - result->Type = ECode::Using; + result->Name = get_cached_string( name, length ); - type->Parent = result; - result->add_entry( type ); + switch ( specifier ) + { + case UsingRegular: + result->Entries = make_code_entries(); + result->Type = ECode::Using; + result->add_entry( type ); + break; + + case UsingNamespace: + result->Type = ECode::Using_Namespace; + break; + } + + result.lock(); return result; } @@ -1116,6 +1328,7 @@ namespace gen Code result = make_code(); + result->Type = Class_Body; result->Entries = make_code_entries(); va_list va; @@ -1126,12 +1339,17 @@ namespace gen switch ( entry->Type ) { + case Enum_Body: + case Execution: + case Function_Body: + case Global_Body: case Namespace: case Namespace_Body: case Parameters: case Specifiers: case Struct_Body: case Typename: + case Using_Namespace: { log_failure("gen::def_class_body: Entry type is not allowed: %s", ECode::str(entry->Type) ); return Code::Invalid; @@ -1153,12 +1371,13 @@ namespace gen while ( num--, num > 0 ); va_end(va); + result.lock(); return result; } Code def_enum_body( s32 num, ... ) { - + not_implemented( def_enum_body ); } Code def_function_body( s32 num, ... ) @@ -1189,7 +1408,15 @@ namespace gen switch ( entry->Type ) { + case Access_Public: + case Access_Protected: + case Access_Private: + case Class_Body: + case Enum_Body: + case Friend: + case Function_Body: case Function_FwdDecl: + case Global_Body: case Namespace: case Namespace_Body: case Operator: @@ -1245,7 +1472,15 @@ namespace gen switch ( entry->Type ) { + case Access_Public: + case Access_Protected: + case Access_Private: + case Class_Body: + case Enum_Body: + case Friend: + case Function_Body: case Function_FwdDecl: + case Global_Body: case Namespace: case Namespace_Body: case Operator: @@ -1271,7 +1506,61 @@ namespace gen Code def_global_body( s32 num, ... ) { + using namespace ECode; + if ( num <= 0 ) + { + log_failure("gen::def_global_body: num cannot zero or neg"); + return Code::Invalid; + } + + Code result = make_code(); + + array_init( result->Entries, g_allocator ); + + va_list va; + va_start(va, num); + do + { + Code entry = va_arg(va, Code); + + if ( ! entry ) + { + log_failure("gen::def_global_body: Provided an invalid entry!"); + return Code::Invalid; + } + + switch ( entry->Type ) + { + case Access_Public: + case Access_Protected: + case Access_Private: + case Class_Body: + case Enum_Body: + case Execution: + case Friend: + case Function_Body: + case Global_Body: + case Namespace_Body: + case Parameters: + case Specifiers: + case Struct_Body: + case Typename: + { + log_failure("gen::def_global_body: Entry type is not allowed: %s", entry->type_str() ); + return Code::Invalid; + } + + default: + break; + } + + result->add_entry( entry ); + } + while ( num--, num > 0 ); + va_end(va); + + return result; } Code def_namespace_body( s32 num, ... ) @@ -1338,7 +1627,7 @@ namespace gen char const* name = va_arg(va, char const*); s32 name_length = strnlen(name, MaxNameLength); - result->Name = cached_string( name, name_length ); + result->Name = get_cached_string( name, name_length ); array_init( result->Entries, g_allocator ); @@ -1352,15 +1641,14 @@ namespace gen while( num -= 2, num && num % 2 == 0 ) { - type = va_arg(va, Code); - + type = va_arg(va, Code); + name_length = va_arg(va, u32); name = va_arg(va, char const*); - name_length = strnlen(name, MaxNameLength); Code param = make_code(); param->Type = Parameters; - param->Name = cached_string(name, name_length); + param->Name = get_cached_string(name, name_length); array_init( param->Entries, StaticData::Allocator_CodePool ); @@ -1381,11 +1669,6 @@ namespace gen return result; } - Code def_params_macro ( s32 num, ... ) - { - - } - Code def_specifiers( s32 num, ... ) { if ( num <= 0 ) @@ -1400,8 +1683,8 @@ namespace gen do_once_end Code - result = make_code(); - result->Type = ECode::Specifiers; + result = make_code(); + result->Type = ECode::Specifiers; String crafted = string_make( arena_allocator( & str_arena ), "" ); @@ -1427,7 +1710,7 @@ namespace gen while ( --num, num ); va_end(va); - result->Content = cached_string( crafted, string_length( crafted ) ); + result->Content = get_cached_string( crafted, string_length( crafted ) ); arena_free( & str_arena ); @@ -1438,9 +1721,9 @@ namespace gen { using namespace ECode; - if ( num == 0 ) + if ( num <= 0 ) { - log_failure("gen::def_struct_body: num cannot be zero"); + log_failure("gen::def_struct_body: num must be at least 1"); return Code::Invalid; } @@ -1456,12 +1739,17 @@ namespace gen switch ( entry->Type ) { + case Enum_Body: + case Execution: + case Function_Body: + case Global_Body: case Namespace: case Namespace_Body: case Parameters: case Specifiers: case Struct_Body: case Typename: + case Using_Namespace: { log_failure("gen::def_struct_body: Entry type is not allowed: %s", ECode::str(entry->Type) ); return Code::Invalid; @@ -1490,12 +1778,12 @@ namespace gen # pragma region Incremetnal Constructors Code make_class( s32 length, char const* name, Code parent, Code specifiers ) { - + not_implemented( make_class ); } Code make_enum( s32 length, char const* name, Code type, EnumT specifier ) { - + not_implemented( make_class ); } Code make_function( s32 length, char const* name @@ -1528,7 +1816,7 @@ namespace gen Code result = make_code(); - result->Name = string_make( g_allocator, name ); + result->Name = get_cached_string( name, length ); result->Type = Function; result->Entries = make_code_entries(); @@ -1558,7 +1846,7 @@ namespace gen Code result = make_code(); result->Type = ECode::Global_Body; - result->Name = cached_string( name, length ); + result->Name = get_cached_string( name, length ); result->Entries = make_code_entries(); // Making body at entry 0; @@ -1569,22 +1857,22 @@ namespace gen Code make_namespace( s32 length, char const* name, Code parent, Code specifiers ) { - + not_implemented( make_namespace ); } Code make_operator( OperatorT op, Code params, Code ret_type, Code specifiers ) { - + not_implemented( make_operator ); } Code make_params() { - + not_implemented( make_params ); } Code make_specifiers() { - + not_implemented( make_specifiers ); } Code make_struct( s32 length, char const* name, Code parent, Code specifiers ) @@ -1608,7 +1896,7 @@ namespace gen Code result = make_code(); result->Type = Struct; - result->Name = string_make( g_allocator, name ); + result->Name = get_cached_string( name, length ); result->Entries = make_code_entries(); Code @@ -1631,22 +1919,27 @@ namespace gen # pragma region Parsing Constructors Code parse_class( s32 length, char const* def ) { - + not_implemented( parse_class ); } Code parse_enum( s32 length, char const* def ) { + not_implemented( parse_enum ); + } + Code parse_execution( s32 length, char const* exec_def ) + { + not_implemented( parse_execution ); } Code parse_friend( s32 length, char const* def ) { - + not_implemented( parse_friend ); } Code parse_global_body( s32 length, char const* def ) { - + not_implemented( parse_global_body ); } Code parse_function( s32 length, char const* def ) @@ -1659,9 +1952,7 @@ namespace gen Arena mem; do_once_start - { arena_init_from_allocator( & mem, heap(), kilobytes( 10 ) ); - } do_once_end // Pretty sure its impossible to have more than this. @@ -1782,20 +2073,21 @@ namespace gen { } + + not_implemented( parse_function ); } Code specifiers = def_specifiers( num_specifiers, specs_found ); Code params = make_code(); - Code ret_type = def_type( ret_type_str ); + Code ret_type = def_type( ret_length, ret_type_str ); Code body = untyped_str( body_length, body_str ); Code - result = make_code(); - result->Name = cached_string( name, name_length ); - result->Type = ECode::Function; - - array_init( result->Entries, g_allocator ); + result = make_code(); + result->Name = get_cached_string( name, name_length ); + result->Type = ECode::Function; + result->Entries = make_code_entries(); result->add_entry( body ); @@ -1813,12 +2105,12 @@ namespace gen Code parse_namespace( s32 length, char const* def ) { - + not_implemented( parse_namespace ); } Code parse_operator( s32 length, char const* def ) { - + not_implemented( parse_operator ); } Code parse_struct( s32 length, char const* def ) @@ -1836,87 +2128,87 @@ namespace gen // Making all significant tokens have a max length of 128 for this parser. ct sw LengthID = 128; - char const name[LengthID] { 0 }; + char const name [LengthID] { 0 }; char const parent[LengthID] { 0 }; } Code parse_variable( s32 length, char const* def ) { - + not_implemented( parse_variable ); } Code parse_type( s32 length, char const* def ) { - + not_implemented( parse_type ); } Code parse_typdef( s32 length, char const* def ) { - + not_implemented( parse_typedef ); } Code parse_using( s32 length, char const* def ) { - + not_implemented( parse_using ); } - s32 parse_classes ( s32 length, char const* class_defs, Code* out_class_codes ) + s32 parse_classes( s32 length, char const* class_defs, Code* out_class_codes ) { - + not_implemented( parse_classes ); } - s32 parse_enums ( s32 length, char const* enum_defs, Code* out_enum_codes ) + s32 parse_enums( s32 length, char const* enum_defs, Code* out_enum_codes ) { - + not_implemented( parse_enums ); } - s32 parse_friends ( s32 length, char const* friend_defs, Code* out_friend_codes ) + s32 parse_friends( s32 length, char const* friend_defs, Code* out_friend_codes ) { - + not_implemented( parse_friends ); } s32 parse_functions ( s32 length, char const* fn_defs, Code* out_fn_codes ) { - + not_implemented( parse_functions ); } s32 parse_namespaces( s32 length, char const* namespace_defs, Code* out_namespaces_codes ) { - + not_implemented( parse_namespaces ); } - s32 parse_operators ( s32 length, char const* operator_defs, Code* out_operator_codes ) + s32 parse_operators( s32 length, char const* operator_defs, Code* out_operator_codes ) { - + not_implemented( parse_operators ); } - s32 parse_structs ( s32 length, char const* struct_defs, Code* out_struct_codes ) + s32 parse_structs( s32 length, char const* struct_defs, Code* out_struct_codes ) { - + not_implemented( parse_structs ); } - s32 parse_variables ( s32 length, char const* vars_def, Code* out_var_codes ) + s32 parse_variables( s32 length, char const* vars_def, Code* out_var_codes ) { - + not_implemented( parse_variables ); } - s32 parse_typedefs ( s32 length, char const* typedef_def, Code* out_typedef_codes ) + s32 parse_typedefs( s32 length, char const* typedef_def, Code* out_typedef_codes ) { - + not_implemented( parse_typedefs ); } - s32 parse_usings ( s32 length, char const* usings_def, Code* out_using_codes ) + s32 parse_usings( s32 length, char const* usings_def, Code* out_using_codes ) { - + not_implemented( parse_usings ); } # pragma endregion Parsing Constructors # pragma region Untyped Constructors - Code untyped_str(char const* str) + Code untyped_str( s32 length, char const* str ) { Code result = make_code(); - result->Name = string_make( g_allocator, str ); + result->Name = get_cached_string( str, length ); result->Type = ECode::Untyped; result->Content = result->Name; @@ -1930,14 +2222,14 @@ namespace gen va_list va; va_start(va, fmt); - snprintf_va(buf, ZPL_PRINTF_MAXLEN, fmt, va); + sw length = snprintf_va(buf, ZPL_PRINTF_MAXLEN, fmt, va); va_end(va); Code result = make_code(); - result->Name = string_make( g_allocator, fmt ); + result->Name = get_cached_string( fmt, strnlen(fmt, MaxNameLength) ); result->Type = ECode::Untyped; - result->Content = string_make( g_allocator, buf ); + result->Content = get_cached_string( buf, length ); return result; } @@ -1949,14 +2241,14 @@ namespace gen va_list va; va_start(va, fmt); - token_fmt_va(buf, ZPL_PRINTF_MAXLEN, fmt, num_tokens, va); + sw length = token_fmt_va(buf, ZPL_PRINTF_MAXLEN, fmt, num_tokens, va); va_end(va); Code - result = make_code(); - result->Name = string_make( g_allocator, fmt ); - result->Type = ECode::Untyped; - result->Content = string_make( g_allocator, buf ); + result = make_code(); + result->Name = get_cached_string( fmt, strnlen(fmt, MaxNameLength) ); + result->Type = ECode::Untyped; + result->Content = get_cached_string( buf, length ); result.lock(); @@ -1966,7 +2258,6 @@ namespace gen #pragma endregion Gen Interface #pragma region Builder - void Builder::print( Code code ) { Buffer = string_append_fmt( Buffer, "%s\n\n", code->to_string() ); @@ -2005,4 +2296,5 @@ namespace gen #pragma region Scanner #pragma endregion Scanner } +// end gentime #endif diff --git a/project/gen.hpp b/project/gen.hpp index 2af254c..f2c2e53 100644 --- a/project/gen.hpp +++ b/project/gen.hpp @@ -11,45 +11,66 @@ ### *WHAT IS NOT PROVIDED* - * Macro or template generation : This library is to avoid those, adding support for them adds unnecessary complexity. - If you desire define them outside the gen_time scopes. - * Expression validation : Execution expressions are defined using the untyped string API. - There is no parse API for validating expressions (possibly will add in the future) - * Modern C++ (STL library) features - * Modern C++ RTTI : This is kinda covered with the last point, but just wanted to emphasize. + This library aims to be used in a "orthodox" or minimal C++ workspace. - Exceptions brought in from "Modern C++": + * Macro or template generation : This library is to avoid those, adding support for them adds unnecessary complexity. + * Vendor provided dynamic dispatch (virtuals) : Roll your own, this library might roll its own vtable/interface generation helpers in the future. + * RTTI : This is kinda covered with the last point, but just wanted to emphasize. + * Exceptions : Most fo the + * Execution statement validation : Execution expressions are defined using the untyped string API. + + Keywords in from "Modern C++": Specifiers: - * consteval - * constinit - * explicit - * export - * noexcept - * import - * final - * module - * override - * && - * virtual + * constexpr : Great to store compile-time constants, (easier to garanteed when emitted from gentime) + * consteval : Technically fine so long as templates are not used. Need to make sure to execute in moderation. + * constinit : Better than constexpr at doing its job, however, its only c++ 20. + * export : Useful if c++ modules ever come around to actually being usable. + * import : ^^ + * module : ^^ - These features are in as they are just specifiers and aren't hard to implement seralization or validation. + These features are in as they either are not horrible when used conservatively or are a performance benefit (modules). + + When it comes to expressions: + There is no parse API for validating expressions (possibly will add in the future). + This reason there isn't one: thats where the can of worms open for parsing validation. + For most metaprogramming (espcially for c/c++), expression validation is not necessary, it can be done by the compiler for the runtime program. + Most of the time, the critical complex metaprogramming conundrums are actaully producing the frame of abstractions around the expressions. + Thus its not very much a priority to add such a level of complexity to the library when there isn't a high reward or need for it. + + To further this point, lets say you do have an error with an expressions composition. + It will either be caught by the c++ compiler when compiling the target program, or at runtime for the program. + + * If its not caught by the compiler, the only downside is the error appers on the generated function. + Those with knowledge of how that definition was generated know where to find the code that inlined that expression in that file for that definition. + * If its caught at runtime. The expression will be shown in a stack trace if debug symbols are enabled in the generated function body. + Yet again those with knowledge of how that definition was generated know where to find the code that inlined that expression. + + In both these cases will get objectively better debug information than you would normally get on most c++ compilers with complex macros or templates. + + ### The Data & Interface: The AST is managed by the library and provided the user via its interface prodedures. Notes: * The allocator definitions used are exposed to the user incase they want to dictate memory usage + * You'll find the memory handling in `init`, `gen_string_allocator`, `get_cached_string`, `make_code`, and `make_code_entries`. * ASTs are wrapped for the user in a Code struct which essentially a warpper for a AST* type. * Both AST and Code have member symbols but their data layout is enforced to be POD types. + * This library treats memory failures as fatal. + * The default setup assumes large definition sets may be provided to bodies so AST::Entires are dynamic arrays. + * They're allocated to arenas currently and are pretty wasteful if they go over their reserve size (its never recycled). + * Most likely will need to implement a dynamic-sized bucket allocation strategy for the entry arrays if memory is getting stressed. + * Otherwise if you are using fixed size entries and your definitions are under 128~512 entries for the body, you may be better of with a fixed-sized array. Data layout of AST struct: AST* Parent; - string_const Name; - string_const Comment; + CachedString Name; + CachedString Comment; union { array(AST*) Entries; - string_const Content; + CachedString Content; }; CodeT Type; OperatorT Op; @@ -61,6 +82,19 @@ ASTs can be set to readonly by calling Code's lock() member function. Adding comments is always available even if the AST is set to readonly. + Data Notes: + + * The allocator definitions used are exposed to the user incase they want to dictate memory usage + * You'll find the memory handling in `init`, `gen_string_allocator`, `get_cached_string`, `make_code`, and `make_code_entries`. + * ASTs are wrapped for the user in a Code struct which essentially a warpper for a AST* type. + * Both AST and Code have member symbols but their data layout is enforced to be POD types. + * This library treats memory failures as fatal. + * The default setup assumes large definition sets may be provided to bodies so AST::Entires are dynamic arrays. + * They're allocated to arenas currently and are pretty wasteful if they go over their reserve size (its never recycled). + * Most likely will need to implement a dynamic-sized bucket allocation strategy for the entry arrays if memory is getting stressed. + * Otherwise if you are using fixed size entries and your definitions are under 128~512 entries for the body, you may be better of with a fixed-sized array. + * Strings are stored in their own set of arenas. AST constructors use cached strings for names, and content. + ### There are four sets of interfaces for Code AST generation the library provides * Upfront @@ -77,7 +111,7 @@ * def_class * def_enum - * def_enum_class + * def_execution NOTE: This is equivalent to untyped_str, except that its intended for use only in execution scopes. * def_friend * def_function * def_namespace @@ -318,19 +352,19 @@ #include "Bloat.hpp" // Temporarily here for debugging purposes. -#define gen_time +#define gentime #define GEN_BAN_CPP_TEMPLATES -#define GEN_ENFORCE_READONLY_AST #define GEN_DEFINE_DSL #define GEN_DEFINE_LIBRARY_CODE_CONSTANTS -#define GEN_USE_FATAL +// #define GEN_DONT_USE_FATAL +#define GEN_ENFORCE_READONLY_AST #define GEN_FEATURE_EDITOR #define GEN_FEATURE_SCANNER -#ifdef gen_time +#ifdef gentime namespace gen { using LogFailType = sw(*)(char const*, ...); @@ -339,10 +373,12 @@ namespace gen # define template static_assert("Templates are banned within gen_time scope blocks") # endif -# ifdef GEN_USE_FATAL - ct LogFailType log_failure = fatal; + // By default this library will either crash or exit if an error is detected while generating codes. + // Even if set to not use fatal, fatal will still be used for memory failures as the library is unusable when they occur. +# ifdef GEN_DONT_USE_FATAL + ct LogFailType log_failure = log_fmt; # else - ct LogFailType log_failure = log_fmt; + ct LogFailType log_failure = fatal; # endif namespace ECode @@ -350,8 +386,8 @@ namespace gen # define Define_Types \ Entry( Untyped ) \ Entry( Access_Public ) \ - Entry( Access_Private ) \ Entry( Access_Protected ) \ + Entry( Access_Private ) \ Entry( Class ) \ Entry( Class_FwdDecl ) \ Entry( Class_Body ) \ @@ -360,6 +396,7 @@ namespace gen Entry( Enum_Body ) \ Entry( Enum_Class ) \ Entry( Enum_Class_FwdDecl ) \ + Entry( Execution ) \ Entry( Friend ) \ Entry( Function ) \ Entry( Function_FwdDecl ) \ @@ -377,7 +414,8 @@ namespace gen Entry( Variable ) \ Entry( Typedef ) \ Entry( Typename ) \ - Entry( Using ) + Entry( Using ) \ + Entry( Using_Namespace ) enum Type : u32 { @@ -484,6 +522,9 @@ namespace gen inline char const* str( Type op ) { + using something = u8; + typedef u8 another; + local_persist char const* lookup[ Num_Ops ] = { # define Entry( Type, Token ) txt(Token), @@ -704,11 +745,11 @@ namespace gen # define Using_Code_POD \ AST* Parent; \ - string_const Name; \ - string_const Comment; \ + StringCached Name; \ + StringCached Comment; \ union { \ Array(AST*) Entries; \ - string_const Content; \ + StringCached Content; \ }; \ CodeT Type; \ OperatorT Op; \ @@ -785,7 +826,7 @@ namespace gen forceinline operator bool() const { - return ast && ast->is_invalid(); + return ast; } bool operator ==( Code other ) const @@ -848,14 +889,12 @@ namespace gen ct Code NoCode = { nullptr }; // extern const Code InvalidCode; - /* - Implements basic string interning. Data structure is based off the ZPL Hashtable. - */ + // Implements basic string interning. Data structure is based off the ZPL Hashtable. ZPL_TABLE_DECLARE( ZPL_EXTERN, StringTable, str_tbl_, String ); // Represents strings cached with the string table. // Should never be modified, if changed string is desired, cache_string( str ) another. - using string_const = char const*; + using StringCached = char const*; /* Type Table: Used to store Typename ASTs. Types are registered by their string literal value. @@ -868,21 +907,18 @@ namespace gen #pragma endregion Data Structures #pragma region Gen Interface - /* - Initialize the library. - This currently just initializes the CodePool. - */ + // Initialize the library. + // This currently just initializes the CodePool. void init(); // Use this only if you know you generated the code you needed to a file. // And rather get rid of current code asts instead of growing the pool memory. + // This generally can be done everytime a file is generated void clear_code_pool(); - /* - Used internally to retrive or make string allocations. - Strings are stored in a series of string arenas of fixed size (SizePer_StringArena) - */ - string_const cached_string( char const* cstr, s32 length ); + // Used internally to retrive or make string allocations. + // Strings are stored in a series of string arenas of fixed size (SizePer_StringArena) + StringCached get_cached_string( char const* cstr, s32 length ); /* This provides a fresh Code AST. @@ -891,51 +927,39 @@ namespace gen */ Code make_code(); - /* - This provides a fresh Code AST array for the entries field of the AST. - This is done separately from the regular CodePool allocator. - */ + // This provides a fresh Code AST array for the entries field of the AST. + // This is done separately from the regular CodePool allocator. Array(AST*) make_code_entries(); // Set these before calling gen's init() procedure. + // Data - void set_allocator_code_pool ( AllocatorInfo pool_allocator ); - void set_allocator_string_arena( AllocatorInfo string_allocator ); - void set_allocator_string_table( AllocatorInfo string_allocator ); - void set_allocator_type_table ( AllocatorInfo type_reg_allocator ); - - void set_init_reserve_code_pool ( sw size ); - void set_init_reserve_code_entries_pool( sw size ); - void set_init_reserve_string_table ( sw size ); - void set_init_reserve_type_table ( sw size ); - - void set_size_string_arena( sw size ); + void set_allocator_data_arrays ( AllocatorInfo data_array_allocator ); + void set_allocator_code_pool ( AllocatorInfo pool_allocator ); + void set_allocator_code_enries_arena( AllocatorInfo pool_allocator ); + void set_allocator_string_arena ( AllocatorInfo string_allocator ); + void set_allocator_string_table ( AllocatorInfo string_allocator ); + void set_allocator_type_table ( AllocatorInfo type_reg_allocator ); # pragma region Upfront - Code def_class ( char const* name, Code parent = NoCode, Code specifiers = NoCode, Code body = NoCode ); Code def_class ( s32 length, char const* name, Code parent = NoCode, Code specifiers = NoCode, Code body = NoCode ); - Code def_enum ( char const* name, Code type = NoCode, EnumT specifier = EnumRegular, Code body = NoCode); Code def_enum ( s32 length, char const* name, Code type = NoCode, EnumT specifier = EnumRegular, Code body = NoCode ); + + Code def_execution ( Code untyped_code ); + Code def_friend ( Code symbol ); - Code def_function ( char const* name, Code params = NoCode, Code ret_type = NoCode, Code specifiers = NoCode, Code body = NoCode ); Code def_function ( s32 length, char const* name, Code params = NoCode, Code ret_type = NoCode, Code specifiers = NoCode, Code body = NoCode ); - Code def_namespace ( char const* name, Code body ); Code def_namespace ( s32 length, char const* name, Code body ); Code def_operator ( OperatorT op, Code params = NoCode, Code ret_type = NoCode, Code specifiers = NoCode, Code body = NoCode ); - Code def_param ( Code type, char const* name ); Code def_param ( Code type, s32 length, char const* name ); Code def_specifier ( SpecifierT specifier ); - Code def_struct ( char const* name, Code parent = NoCode, Code specifiers = NoCode, Code body = NoCode ); Code def_struct ( s32 length, char const* name, Code parent = NoCode, Code specifiers = NoCode, Code body = NoCode ); - Code def_type ( char const* name, Code specifiers = NoCode ); Code def_type ( s32 length, char const* name, Code specifiers = NoCode ); - Code def_using ( char const* name, Code type = NoCode, UsingT specifier = UsingRegular ); Code def_using ( s32 length, char const* name, Code type = NoCode, UsingT specifier = UsingRegular ); - Code def_variable ( Code type, char const* name, Code value = NoCode, Code specifiers = NoCode ); Code def_variable ( Code type, s32 length, char const* name, Code value = NoCode, Code specifiers = NoCode ); Code def_class_body ( s32 num, ... ); @@ -954,26 +978,21 @@ namespace gen # pragma endregion Upfront # pragma region Incremental - Code make_class ( char const* name, Code parent = NoCode, Code specifiers = NoCode ); Code make_class ( s32 length, char const* name, Code parent = NoCode, Code specifiers = NoCode ); - Code make_enum ( char const* name, Code type = NoCode, EnumT specifier = EnumRegular ); Code make_enum ( s32 length, char const* name, Code type = NoCode, EnumT specifier = EnumRegular ); - Code make_function ( char const* name, Code params = NoCode, Code ret_type = NoCode, Code specifiers = NoCode ); - Code make_function ( s32 length, char const* name, Code params = NoCode, Code ret_type = NoCode, Code specifiers = NoCode ); - Code make_global_body ( char const* name = "", s32 num = 0, ... ); + Code make_function ( s32 length, char const* name, Code params = NoCode, Code ret_type = NoCode, Code specifiers = NoCode ); Code make_global_body ( s32 length = 1, char const* name = "", s32 num = 0, ... ); - Code make_namespace ( char const* name ); Code make_namespace ( s32 length, char const* name ); - Code make_operator ( OperatorT op, Code params = NoCode, Code ret_type = NoCode, Code specifiers = NoCode ); + Code make_operator ( OperatorT op, Code params = NoCode, Code ret_type = NoCode, Code specifiers = NoCode ); Code make_params (); Code make_specifiers (); - Code make_struct ( char const* name, Code parent = NoCode, Code specifiers = NoCode ); - Code make_struct ( s32 length, char const* name, Code parent = NoCode, Code specifiers = NoCode ); + Code make_struct ( s32 length, char const* name, Code parent = NoCode, Code specifiers = NoCode ); # pragma endregion Incremental # pragma region Parsing Code parse_class ( s32 length, char const* class_def ); Code parse_enum ( s32 length, char const* enum_def ); + Code parse_execution ( s32 length, char const* exec_def ); Code parse_friend ( s32 length, char const* friend_def ); Code parse_function ( s32 length, char const* fn_def ); Code parse_global_body( s32 length, char const* body_def ); @@ -998,7 +1017,6 @@ namespace gen # pragma endregion Parsing # pragma region Untyped text - Code untyped_str ( char const* str ); Code untyped_str ( s32 length, char const* str); Code untyped_fmt ( char const* fmt, ... ); Code untyped_token_fmt( char const* fmt, s32 num_tokens, ... ); @@ -1030,7 +1048,7 @@ namespace gen struct SymbolInfo { - string_const File; + StringCached File; char const* Marker; Code Signature; }; @@ -1061,7 +1079,7 @@ namespace gen struct Receipt { - string_const File; + StringCached File; Code Found; Code Written; bool Result; @@ -1099,7 +1117,7 @@ namespace gen struct Receipt { - string_const File; + StringCached File; Code Defintion; bool Result; }; @@ -1132,7 +1150,7 @@ namespace gen // Used by the DSL but can also be used without it. # define type_ns( Name_ ) t_##Name_ -// Convienence for defining any name used if desring to use library +// Convienence for defining any name used with the gen interface. // Lets you provide the length and string literal to the functions without the need for the DSL. # define name( Id_ ) txt_n_len( Id_ ) @@ -1254,9 +1272,7 @@ namespace gen #ifdef GEN_DEFINE_LIBRARY_CODE_CONSTANTS namespace gen { - // Predefined typename codes. - // These are not set until gen::init is called. - // This just preloads a bunch of Code types into the code pool. + // Predefined typename codes. Are set to readonly and are setup during gen::init() extern Code type_ns( void ); @@ -1284,10 +1300,25 @@ namespace gen namespace gen { + // These constexprs are used for allocation heavior of data structurs + // or string handling while constructing or serializing. + // Change them to suit your needs. + + ct s32 InitSize_DataArrays = 16; + ct s32 InitSize_StringTable = megabytes(4); + ct s32 InitSize_TypeTable = megabytes(4); + + ct s32 CodePool_NumBlocks = 4096; + ct s32 CodeEntiresPool_NumBlocks = 4096; + ct s32 SizePer_CodeEntriresArena = megabytes(16); + ct s32 SizePer_StringArena = megabytes(32); + ct s32 MaxNameLength = 128; ct s32 MaxUntypedStrLength = kilobytes(640); ct s32 StringTable_MaxHashLength = kilobytes(1); + // Predefined Codes. Are set to readonly and are setup during gen::init() + extern Code access_public; extern Code access_protected; extern Code access_private; @@ -1300,100 +1331,5 @@ namespace gen } #pragma endregion Constants -#pragma region Gen Interface Inlines -namespace gen -{ - forceinline - Code def_class( char const* name, Code parent, Code specifiers, Code body ) - { - return def_class( strnlen( name, MaxNameLength ), name, parent, specifiers, body ); - } - - forceinline - Code def_enum( char const* name, Code type, EnumT specifier, Code body ) - { - return def_enum( strnlen( name, MaxNameLength ), name, type, specifier, body ); - } - - forceinline - Code def_function( char const* name, Code params, Code ret_type, Code specifiers, Code body ) - { - return def_function( strnlen( name, MaxNameLength), name, params, ret_type, specifiers, body ); - } - - forceinline - Code def_namespace( char const* name, Code body ) - { - return def_namespace( strnlen( name, MaxNameLength), name, body ); - } - - forceinline - Code def_param( Code type, char const* name ) - { - return def_param( type, strnlen( name, MaxNameLength ), name ); - } - - forceinline - Code def_struct( char const* name, Code parent, Code specifiers, Code body ) - { - return def_struct( strnlen( name, MaxNameLength), name, parent, specifiers, body ); - } - - forceinline - Code def_type( char const* name, Code specifiers ) - { - return def_type( strnlen( name, MaxNameLength ), name, specifiers ); - } - - forceinline - Code def_using( char const* name, Code type, UsingT specifier ) - { - return def_using( strnlen( name, MaxNameLength ), name, type, specifier ); - } - - forceinline - Code def_variable( Code type, char const* name, Code value, Code specifiers ) - { - return def_variable( type, strnlen(name, MaxNameLength ), name, value, specifiers ); - } - - forceinline - Code make_class( char const* name, Code parent, Code specifiers ) - { - return make_class( strnlen(name, MaxNameLength), name, parent, specifiers ); - } - - forceinline - Code make_enum( char const* name, Code type, Code specifiers ) - { - return make_struct( strnlen(name, MaxNameLength), name, type, specifiers ); - } - - forceinline - Code make_function( char const* name, Code params, Code ret_type, Code specifiers ) - { - return make_function( strnlen(name, MaxNameLength), name, params, ret_type, specifiers ); - } - - forceinline - Code make_namespace( char const* name ) - { - return make_namespace( strnlen( name, MaxNameLength ), name ); - } - - forceinline - Code make_struct( char const* name, Code parent, Code specifiers ) - { - return make_struct( strnlen(name, MaxNameLength), name, parent, specifiers ); - } - - forceinline - Code untyped_str( char const* str ) - { - return untyped_str( strnlen( str, MaxUntypedStrLength ), str ); - } -} -#pragma endregion Gen Interface Inlines - // end: gen_time #endif diff --git a/test/test.cpp b/test/test.cpp index 18d0fdf..d615abb 100644 --- a/test/test.cpp +++ b/test/test.cpp @@ -19,7 +19,7 @@ int gen_main() #endif -#ifdef runtime +#ifdef comptime int main() { return 0;