Update readmes. Some minor changes to the API.

Readme.md

@@ -2,7 +2,7 @@
 
 An attempt at simple staged metaprogramming for c/c++.
 
-The library is mostly a compositon of code element constructors.
+The library API is a compositon of code element constructors.
 These build up a code AST to then serialize with a file builder.
 
 Intended for small-to midsized projects.
@@ -11,11 +11,11 @@ Intended for small-to midsized projects.
 
 * [Notes](#notes)
 * [Usage](#usage)
-* [Gen's DSL](#gens-dsl)
 * [Building](#notes)
 * [Outline](#outline)
 * [What is not provided](#what-is-not-provided)
 * [The four constructors](#there-are-four-sets-of-interfaces-for-code-ast-generation-the-library-provides)
+* [Predefined Codes](#predefined-codes)
 * [Code generation and modification](#code-generation-and-modification)
 * [On multithreading](#on-multi-threading)
 * [Extending the library](#extending-the-library)
@@ -29,12 +29,14 @@ Version 1 will have C and a subset of C++ features available to it.
 I will generate with this library a C99 or 11 variant when Version 1 is complete.
 A single-header version will also be generated.
 
-The size target of this library is to stay under 5-6k sloc (data & interface code).
+The size target is to stay under 5-6k sloc (data & interface code).
 With the dependency code being under 10000 sloc. (Containers, Memory, String handling, Language bloat)
 
 Any dependencies from the zpl library will be exposed manually with using declarations into global scope.
 They will be removed when the library is feature complete for version 1 (zero dependencies milestone).
 
+***The editor and scanner will NOT be implemented by version 1. They require alot of code and the focus for version 1 is to have a robust constructor API and builder, witch a wide suite of usage examples in the tests for the project.***
+
 ## Usage
 
 A metaprogram is built to generate files before the main program is built. We'll term runtime for this program as `gen_time`. The metaprogram's core implementation are within `gen.hpp` and `gen.cpp` in the project directory.
@@ -67,8 +69,8 @@ u32 gen_main()
 
 ```
 
-The design a constructive builder of a sort of AST for the code to generate.
+The design uses a constructive builder sort of AST for the code to generate.
-The user is given `Code` typed objects that are used to build up the AST.
+The user is given `Code` objects that are used to build up the AST.
 
 Example using each construction interface:
 
@@ -125,7 +127,7 @@ Code header = parse_struct( code(
 
 Parse will automatically generate any types that have not been used previously.
 
-### Undtyped
+### Untyped
 
 ```cpp
 Code header = untyped_str( R("
@@ -161,10 +163,10 @@ An example of building is provided in the test directory.
 There are two meson build files the one within test is the program's build specification.
 The other one in the gen directory within test is the metaprogram's build specification.
 
-Both of them build the same source file: `test.cpp`. The only differences between them is that gen need a different relative path to the include directories and defines the macro definition: `gen_time`.
+Both of them build the same source file: `test.cpp`. The only differences are that gen needs a different relative path to the include directories and defines the macro definition: `gen_time`.
 
 This method is setup where all the metaprogram's code are the within the same files as the regular program's code.
-If in your use case, decide to have exclusive separation or partial separation of the metaprogam's code from the program's code files then your build configuration would need to change to reflect that (specifically the sources).
+If in your use case, you decide to have exclusive separation or partial separation of the metaprogam's code from the program's code files, then your build configuration would need to change to reflect that (specifically the sources).
 
 ## Outline
 
@@ -192,7 +194,7 @@ When it comes to expressions:
 There is no support for validating expressions.
 The reason: thats where the can of worms open for parsing validation. This library would most likey more than double in size with that addition alone.
 For most metaprogramming (espcially for C/C++), expression validation is not necessary for metaprogramming, it can be done by the compiler for the runtime program.
-Most of the time, the critical complex metaprogramming conundrums are actaully producing the frame of abstractions around the expressions.
+Most of the time, the critical complex metaprogramming conundrums are producing the frame of abstractions around the expressions.
 Thus its not very much a priority to add such a level of complexity to the library when there isn't a high reward or need for it.
 
 To further this point, lets say you do have an error with an expressions composition.
@@ -207,9 +209,9 @@ In both these cases the user will get objectively better debug information than
 
 ### The Data & Interface
 
-As mentioned in [Usage](#usage), the user is provided Code objects by calling the constructor functions to generate them or find existing matches.
+As mentioned in [Usage](#usage), the user is provided Code objects by calling the constructor's functions to generate them or find existing matches.
 
-The AST is managed by the library and provided the user via its interface prodedures.
+The AST is managed by the library and provided the user via its interface.
 However, the user may specificy memory configuration.
 
 Data layout of AST struct:
@@ -217,19 +219,20 @@ Data layout of AST struct:
 ```cpp
 union {
     AST*          ArrStatic[AST::ArrS_Cap];
-    Array(AST*)   Entries;
+    Array(AST*)   ArrDyn;
     StringCached  Content;
     SpecifierT    ArrSpecs[AST::ArrSpecs_Cap];
 };
 AST*              Parent;
 StringCached      Name;
-StringCached      Comment;
 CodeT             Type;
 OperatorT         Op;
+ModuleFlag        ModuleFlags;
+AccessSpec        ParentAccess;
+u32               StaticIndex;
 bool              Readonly;
 bool              DynamicEntries;
-u8                StaticIndex;
+u8                _Align_Pad[2];
-u8                _Align_Pad[6];
 ```
 
 *`CodeT` is a typedef for `ECode::Type` which has an underlying type of `u32`*
@@ -243,16 +246,19 @@ The width dictates how much the static array can hold before it must give way to
 constexpr static
 uw ArrS_Cap =
 (     AST_POD_Size
-    - sizeof(AST*)
+    - sizeof(AST*)         // Parent
-    - sizeof(StringCached) * 2
+    - sizeof(StringCached) // Name
-    - sizeof(CodeT)
+    - sizeof(CodeT) 	   // Type
-    - sizeof(OperatorT)
+    - sizeof(OperatorT)    // Op
-    - sizeof(bool) * 2
+    - sizeof(ModuleFlag)   // ModuleFlags
-    - sizeof(u8) * 7 )
+    - sizeof(AccessSpec)   // ParentAccess
+    - sizeof(u32) 		   // StaticIndex
+    - sizeof(bool) * 2     // Readonly, DynamicEntries
+    - sizeof(u8) * 2 )     // _Align_Pad
 / sizeof(AST*);
 ```
 
-*Ex: If the AST_POD_Size is 256 the capacity of the static array is 26.*
+*Ex: If the AST_POD_Size is 256 the capacity of the static array is 27.*
 
 ASTs can be set to readonly by calling Code's lock() member function.
 Adding comments is always available even if the AST is set to readonly.
@@ -261,7 +267,7 @@ Data Notes:
 
 * The allocator definitions used are exposed to the user incase they want to dictate memory usage
   * You'll find the memory handling in `init`, `gen_string_allocator`, `get_cached_string`, `make_code`, and `make_code_entries`.
-* ASTs are wrapped for the user in a Code struct which essentially a warpper for a AST* type.
+* ASTs are wrapped for the user in a Code struct which is a warpper for a AST* type.
 * Both AST and Code have member symbols but their data layout is enforced to be POD types.
 * This library treats memory failures as fatal.
 * Strings are stored in their own set of arenas. AST constructors use cached strings for names, and content.
@@ -280,11 +286,16 @@ The construction will fail and return InvalidCode otherwise.
 
 Interface :
 
+* def_attributes
+* def_comment
 * def_class
 * def_enum
 * def_execution       NOTE: This is equivalent to untyped_str, except that its intended for use only in execution scopes.
+* def_extern_link
 * def_friend
 * def_function
+* def_include
+* def_module
 * def_namespace
 * def_operator
 * def_param
@@ -292,19 +303,24 @@ Interface :
 * def_specifier
 * def_specifiers
 * def_struct
-* def_variable
 * def_type
 * def_typedef
+* def_union
 * def_using
+* def_using_namespace
+* def_variable
 
 Bodies:
 
 * def_class_body
 * def_enum_body
+* def_export_body
+* def_extern_link_body
 * def_function_body   NOTE: Use this for operator bodies as well.
 * def_global_body
 * def_namespace_body
 * def_struct_body
+* def_union_body
 
 Usage:
 
@@ -332,7 +348,8 @@ Interface :
 
 * make_class
 * make_enum
-* make_enum_class
+* make_export_body
+* make_extern_linkage
 * make_function
 * make_global_body
 * make_namespace
@@ -340,6 +357,7 @@ Interface :
 * make_params
 * make_specifiers
 * make_struct
+* make_union
 
 Usage:
 
@@ -359,35 +377,23 @@ Interface :
 
 * parse_class
 * parse_enum
+* parse_export_body
+* parse_extern_link
 * parse_friend
 * parse_function
 * parse_global_body
 * parse_namespace
 * parse_operator
 * parse_struct
-* parse_strucs
-* parse_variable
 * parse_type
 * parse_typedef
+* parse_union
 * parse_using
-
+* parse_variable
-Multiple defs:
-
-* parse_classes
-* parse_enums
-* parse_functions
-* parse_namespaces
-* parse_operators
-* parse_variables
-* parse_typedefs
-* parse_usings
 
 The parse API treats any execution scope definitions with no validation and are turned into untyped Code ASTs.
 This includes the assignmetn of variables; due to the library not yet supporting c/c++ expression parsing.
 
-The plural variants provide an array of codes, its up to the user to add them to a body AST
-(they are not auto-added to a body)
-
 Usage:
 
 ```cpp
@@ -411,6 +417,7 @@ Code ASTs are constructed using unvalidated strings.
 
 Interface :
 
+* token_fmt_va
 * token_fmt
 * untyped_str
 * untyped_fmt
@@ -446,6 +453,57 @@ char const* gen_code_str = token_fmt( template, num_tokens, { token_key, token_v
 Code        <name>       = parse_<function name>( gen_code_str );
 ```
 
+## Predefined Codes
+
+The following are provided predefined by the library as they are commonly used:
+
+* `access_public`
+* `access_protected`
+* `access_private`
+* `module_global_fragment`
+* `module_private_fragment`
+* `pragma_once`
+* `spec_const`
+* `spec_consteval`
+* `spec_constexpr`
+* `spec_constinit`
+* `spec_extern_linkage`
+* `spec_inline`
+* `spec_internal_linkage`
+* `spec_local_persist`
+* `spec_mutable`
+* `spec_ptr`
+* `spec_ref`
+* `spec_register`
+* `spec_rvalue`
+* `spec_static_member`
+* `spec_thread_local`
+* `spec_volatile`
+* `spec_type_signed`
+* `spec_type_unsigned`
+* `spec_type_short`
+* `spec_type_long`
+* `type_ns(void)`
+* `type_ns(int)`
+* `type_ns(bool)`
+* `type_ns(char)`
+* `type_ns(wchar_t)`
+
+Optionally the following may be defined if `GEN_DEFINE_LIBRARY_CODE_CONSTANTS` is defined
+
+* `type_ns( s8 )`
+* `type_ns( s16 )`
+* `type_ns( s32 )`
+* `type_ns( s64 )`
+* `type_ns( u8 )`
+* `type_ns( u16 )`
+* `type_ns( u32 )`
+* `type_ns( u64 )`
+* `type_ns( sw )`
+* `type_ns( uw )`
+* `type_ns( f32 )`
+* `type_ns( f64 )`
+
 ## Extent of operator overload validation
 
 The AST and constructors will be able to validate that the arguments provided for the operator type match the expected form:
@@ -539,6 +597,19 @@ Code parse_defines() can emit a custom code AST with Macro_Constant type.
 
 Another would be getting preprocessor or template metaprogramming Codes from Unreal Engine definitions, etc.
 
+The rules for constructing the AST are largely bound the syntax rules for what can be composed with whichever version of C++ your targeting.
+
+The convention you'll see used throughout the API of the library is as follows:
+
+1. Check name or parameters to make sure they are valid for the construction requested
+2. Create a code object using `make_code`.
+3. Populate immediate fields (Name, Type, ModuleFlags, etc)
+4. Populate sub-entires using `add_entry`. If using the default seralization function `to_string`, follow the order at which entires are expected to appear (there is a strong ordering expected).
+
+Names or Content fields are interned strings and thus showed be cached using `get_cached_string` if its desired to preserve that behavior.
+
+`def_operator` is the most sophisitacated constructor as it has multiple permutations of definitions that could be created that are not trivial to determine if valid.
+
 # TODO
 
 * May be in need of a better name, I found a few repos with this same one...
@@ -552,7 +623,3 @@ Another would be getting preprocessor or template metaprogramming Codes from Unr
   * Most likely at least Incremental could possibly be removed in favor of just using the parse constructors.
   * Possible merits are ergonomics for very dynamic generation or performance reasons.
   * They'll most likely stay until its evident that they are not necessary.
-* Review memory handling for the AST, specifically relating to:
-  * Giving type asts a dedicated memory arenas.
-  * Giving specifier definitions a dedicated memory arenas and hashtable lookup.
-  * Possibly adding a dedicated block allocator for the dynamic arrays of AST::Entires.

project/Readme.md

@@ -18,6 +18,7 @@ Organization:
 * ZPL inclusion and selective symbol exposure to global scope.
 * Utility macro definitions used throughout the library.
 * Global memory arena definition
+* StrC and String definitions
 * Token string formatter
 * Formatted and Fatal Logs
 
@@ -52,7 +53,10 @@ Major enum definitions and their associated functions used with the AST data
 
 * `ECode` : Used to tag ASTs by their type
 * `EOperator` : Used to tag operator overloads with thier op type
-* `ESpecifier` : Used with specifier ASTs for all specifiers the user may tag an associated AST with.
+* `ESpecifier` : Used with specifier ASTs for all specifiers the user may tag an associated
+AST with.
+* `AccessSpec` : Used with class and struct ASTs to denote the public, protected, or private fields.
+* `ModuleFlag` : Used with any valid definition that can have export or import related keywords assoicated with it.
 
 #### Data Structures
 
@@ -85,3 +89,17 @@ Inlined functions related to the AST datatype that required forwards for gen int
 
 ## gen.cpp
 
+* Static data defined at the top.
+* AST Body case macros are next.
+* AST implementation follows
+* Gen interface begins with its `init`, `deinit`, etc.. Until `make_code_entires`
+* operator__validate defined, which will be used to verify if an operator definition is constructible.
+* Allocator interface for data arrays, code pool, code entries, string arenas, and string table.
+* Helper macros used throughout the constructor API
+* Upfront constructors, following the same order as shown in the header.
+* Incremental constructors
+* Parsing constructors, it defines its own lexer, and has many helper functions for parsing not exposed through the header.
+* Untyped constructors
+* Builder
+* Editor
+* Scanner

project/gen.cpp

@@ -57,6 +57,8 @@ namespace gen
 	Code module_global_fragment;
 	Code module_private_fragment;
 
+	Code pragma_once;
+
 	Code spec_const;
 	Code spec_consteval;
 	Code spec_constexpr;
@@ -1336,21 +1338,21 @@ namespace gen
 		access_private_write = ccast( Code, access_private );
 		access_private_write = make_code();
 		access_private_write->Type = ECode::Access_Private;
-		access_private_write->Name = get_cached_string( { sizeof("private"), "private" } );
+		access_private_write->Name = get_cached_string( StrC::from("private") );
 		access_private_write.lock();
 
 		Code&
 		access_protected_write = ccast( Code, access_protected );
 		access_protected_write = make_code();
 		access_protected_write->Type = ECode::Access_Protected;
-		access_protected_write->Name = get_cached_string( { sizeof("protected"), "protected" } );
+		access_protected_write->Name = get_cached_string( StrC::from("protected") );
 		access_protected_write.lock();
 
 		Code&
 		access_public_write = ccast( Code, access_public );
 		access_public_write = make_code();
 		access_public_write->Type = ECode::Access_Public;
-		access_public_write->Name = get_cached_string( { sizeof("public"), "public" } );
+		access_public_write->Name = get_cached_string( StrC::from("public") );
 		access_public_write.lock();
 
 		module_global_fragment          = make_code();
@@ -1365,6 +1367,12 @@ namespace gen
 		module_private_fragment->Content = module_private_fragment->Name;
 		module_private_fragment.lock();
 
+		pragma_once          = make_code();
+		pragma_once->Type    = ECode::Untyped;
+		pragma_once->Name    = get_cached_string( StrC::from("#pragma once") );
+		pragma_once->Content = pragma_once->Name;
+		pragma_once.lock();
+
 		Code&
 		spec_local_persist_write = ccast( Code, spec_local_persist );
 		spec_local_persist_write = def_specifiers( 1, ESpecifier::Local_Persist );
@@ -2009,6 +2017,23 @@ namespace gen
 	The largest of the functions is related to operator overload definitions.
 	I decided to validate a good protion of their form and thus the argument processing for is quite a bit.
 */
+	Code def_attributes( StrC content )
+	{
+		if ( content.Len <= 0 || content.Ptr == nullptr )
+		{
+			log_failure( "gen::def_attributes: Invalid attributes provided" );
+			return Code::Invalid;
+		}
+
+		Code
+		result = make_code();
+		result->Type    = ECode::Attributes;
+		result->Name    = get_cached_string( content );
+		result->Content = result->Name;
+
+		result.lock();
+		return result;
+	}
 
 	Code def_comment( StrC content )
 	{
@@ -2028,24 +2053,6 @@ namespace gen
 		return result;
 	}
 
-	Code def_attributes( StrC content )
-	{
-		if ( content.Len <= 0 || content.Ptr == nullptr )
-		{
-			log_failure( "gen::def_attributes: Invalid attributes provided" );
-			return Code::Invalid;
-		}
-
-		Code
-		result = make_code();
-		result->Type    = ECode::Attributes;
-		result->Name    = get_cached_string( content );
-		result->Content = result->Name;
-
-		result.lock();
-		return result;
-	}
-
 	Code def_class( StrC name
 		, Code body
 		, Code parent, AccessSpec parent_access
@@ -3442,7 +3449,7 @@ namespace gen
 		return result;
 	}
 
-	Code make_extern_linkage( StrC name, ModuleFlag mflags )
+	Code make_extern_link( StrC name, ModuleFlag mflags )
 	{
 		using namespace ECode;
 

project/gen.hpp

@@ -556,8 +556,6 @@ namespace gen
 		constexpr static
 		uw ArrSpecs_Cap = ArrS_Cap * (sizeof(AST*) / sizeof(SpecifierT));
 
-		// static_assert( sizeof( AST* ) * ArrS_Cap == sizeof( AST** ) * ArrS_Cap, "Blah" );
-
 	#	define Using_AST_POD                           \
 		union {                                        \
 			AST*          ArrStatic[AST::ArrS_Cap];    \
@@ -778,8 +776,8 @@ namespace gen
 	void set_allocator_type_table       ( AllocatorInfo type_reg_allocator );
 
 #	pragma region Upfront
-	Code def_comment   ( StrC content );
 	Code def_attributes( StrC content );
+	Code def_comment   ( StrC content );
 
 	Code def_class( StrC name
 		, Code body         = NoCode
@@ -819,8 +817,8 @@ namespace gen
 		, Code       attributes = NoCode
 		, ModuleFlag mflags     = ModuleFlag::None );
 
-	Code def_typedef( StrC name, Code type, Code attributes = NoCode, ModuleFlag mflags = ModuleFlag::None );
 	Code def_type   ( StrC name, Code arrayexpr = NoCode, Code specifiers = NoCode );
+	Code def_typedef( StrC name, Code type, Code attributes = NoCode, ModuleFlag mflags = ModuleFlag::None );
 
 	Code def_union( StrC name, Code body, Code attributes = NoCode, ModuleFlag mflags = ModuleFlag::None );
 
@@ -842,10 +840,10 @@ namespace gen
 	Code def_export_body     ( s32 num, Code* codes);
 	Code def_extern_link_body( s32 num, ... );
 	Code def_extern_link_body( s32 num, Code* codes );
-	Code def_global_body     ( s32 num, ... );
-	Code def_global_body     ( s32 num, Code* codes );
 	Code def_function_body   ( s32 num, ... );
 	Code def_function_body   ( s32 num, Code* codes );
+	Code def_global_body     ( s32 num, ... );
+	Code def_global_body     ( s32 num, Code* codes );
 	Code def_namespace_body  ( s32 num, ... );
 	Code def_namespace_body  ( s32 num, Code* codes );
 	Code def_params          ( s32 num, ... );
@@ -869,8 +867,8 @@ namespace gen
 		, Code type       = NoCode, EnumT      specifier = EnumRegular
 		, Code attributes = NoCode, ModuleFlag mflags    = ModuleFlag::None );
 
-	Code make_export_body   ( StrC name = { 1, "" } );
+	Code make_export_body( StrC name = { 1, "" } );
-	Code make_extern_linkage( s32 length,     char const* name, ModuleFlag mflags = ModuleFlag::None );
+	Code make_extern_link( s32 length,     char const* name, ModuleFlag mflags = ModuleFlag::None );
 
 	Code make_function( StrC name
 		, Code       params     = NoCode, Code ret_type   = NoCode
@@ -1138,6 +1136,8 @@ namespace gen
 	extern Code module_global_fragment;
 	extern Code module_private_fragment;
 
+	extern Code pragma_once;
+
 	extern Code spec_const;
 	extern Code spec_consteval;
 	extern Code spec_constexpr;

test/Readme.md

@@ -3,10 +3,11 @@
 The following tests focus on attempting to generate some math, containers, and the memory module of zpl.
 
 Not all the files are written how I would practically use the librarry,
-there is a bunch of conditionally compiled code for testing different aspects of the library.
-
-Specifically, Array.hpp is far more complicated that it would need to be;  
-with at least double the code that it would actually have to define itself.
 
 There will be down the line a proper container, and memory libraries made with this gen library
 once the stress test files are complete.
+
+The test is divided between two major sets of tests: Parsed and Nonparsed.
+
+Parsed uses the parsing api strictly.
+NonParsed only uses the upfront and incremental constructors.