Reorganization of files, refactors, doc updates (WIP)

Removing the gen. namespace from the files for components, dependencies, and file_processors.
They are only necessary if the include directory is transparent, and in my case those are not.

Made a docs directory. I'm offloading information from the main readme to there along with additional informationn I end up elaborating on down the line.
Enum tables were moved to their own directory (project/enums).

Library will not compile for now. Major refactor occuring with parsing related components.

Docs/Parsing.md

@@ -0,0 +1,49 @@
+# Parsing
+
+The library features a naive parser tailored for only what the library needs to construct the supported syntax of C++ into its AST.  
+This parser does not, and should not do the compiler's job. By only supporting this minimal set of features, the parser is kept under 5000 loc.
+
+
+Everything is done in one pass for both the preprocessor directives and the rest of the language.  
+The parser performs no macro expansion as the scope of gencpp feature-set is to only support the preprocessor for the goal of having rudimentary awareness of preprocessor ***conditionals***,  ***defines***, and ***includes***.  
+*(Conditionals and defines are a TODO)*
+
+The keywords supported for the preprocessor are:
+
+* include
+* define
+* if
+* ifdef
+* elif
+* endif
+* undef
+
+Just like with actual preprocessor, each directive # line is considered one preproecessor unit, and will be treated as one Preprocessor AST. *These ASTs will be considered members or entries of braced scope they reside within*.
+All keywords except *include* are suppported as members of a scope for a class/struct, global, or namespace body.  
+
+Any preprocessor definition abuse that changes the syntax of the core language is unsupported and will fail to parse if not kept within an execution scope (function body, or expression assignment).
+
+The parsing implementation supports the following for the user:
+
+```cpp
+CodeClass     parse_class        ( StrC class_def     );
+CodeEnum      parse_enum         ( StrC enum_def      );
+CodeBody      parse_export_body  ( StrC export_def    );
+CodeExtern    parse_extern_link  ( StrC exten_link_def);
+CodeFriend    parse_friend       ( StrC friend_def    );
+CodeFn        parse_function     ( StrC fn_def        );
+CodeBody      parse_global_body  ( StrC body_def      );
+CodeNamespace parse_namespace    ( StrC namespace_def );
+CodeOperator  parse_operator     ( StrC operator_def  );
+CodeOpCast    parse_operator_cast( StrC operator_def  );
+CodeStruct    parse_struct       ( StrC struct_def    );
+CodeTemplate  parse_template     ( StrC template_def  );
+CodeType      parse_type         ( StrC type_def      );
+CodeTypedef   parse_typedef      ( StrC typedef_def   );
+CodeUnion     parse_union        ( StrC union_def     );
+CodeUsing     parse_using        ( StrC using_def     );
+CodeVar       parse_variable     ( StrC var_def       );
+```
+
+***Parsing will aggregate any tokens within a function body or expression statement to an untyped Code AST.***
+

Docs/Readme.md

@@ -0,0 +1,537 @@
+## Documentation
+
+The project has no external dependencies beyond:
+
+* `errno.h`
+* `stat.h`
+* `stdarg.h`
+* `stddef.h`
+* `stdio.h`
+* `copyfile.h` (Mac)
+* `types.h`    (Linux)
+* `unistd.h`   (Linux/Mac)
+* `intrin.h`   (Windows)
+* `io.h`       (Windows with gcc)
+* `windows.h`  (Windows)
+
+Dependencies for the project are wrapped within `GENCPP_ROLL_OWN_DEPENDENCIES` (Defining it will disable them).  
+The majority of the dependency's implementation was derived from the [c-zpl library](https://github.com/zpl-c/zpl).
+
+This library was written in a subset of C++ where the following are not used at all:
+
+* RAII (Constructors/Destructors), lifetimes are managed using named static or regular functions.
+* Language provide dynamic dispatch, RTTI
+* Object-Oriented Inheritance
+* Exceptions
+
+Polymorphic & Member-functions are used as an ergonomic choice, along with a conserative use of operator overloads.  
+There are only 4 template definitions in the entire library. (`Array<Type>`, `Hashtable<Type>`, `swap<Type>`, and `AST/Code::cast<Type>`)
+
+Two generic templated containers are used throughout the library:
+
+* `template< class Type> struct Array`
+* `template< class Type> struct HashTable`
+
+Both Code and AST definitions have a `template< class Type> Code/AST cast()`. Its just an alternative way to explicitly cast to each other.
+
+`template< class Type> swap( Type& a, Type& b)` is used over a macro.
+
+Otherwise the library is free of any templates.
+
+### *WHAT IS NOT PROVIDED*
+
+* Execution statement validation : Execution expressions are defined using the untyped AST.
+  * Lambdas (This naturally means its unsupported)
+* RAII : This needs support for constructors/destructor parsing
+  * Haven't gotten around to yet (its in the github issues)
+
+Keywords kept from "Modern C++":
+
+* constexpr : Great to store compile-time constants.
+* consteval : Technically fine, need to make sure to execute in moderation.
+* constinit : Better than constexpr at doing its job, however, its only c++ 20.
+* export    : Useful if c++ modules ever come around to actually being usable.
+* import    : ^^
+* module    : ^^
+
+When it comes to expressions:
+
+**There is no support for validating expressions.**  
+Its difficult to parse without enough benefits (At the metaprogramming level).  
+
+When it comes to templates:
+
+**Only trivial template support is provided.**  
+The intention is for only simple, non-recursive substitution.  
+The parameters of the template are treated like regular parameter AST entries.  
+This means that the typename entry for the parameter AST would be either:
+
+* `class`
+* `typename`
+* A fundamental type, function, or pointer type.
+
+Anything beyond this usage is not supported by parse_template for arguments (at least not intentionally).  
+Use at your own mental peril.
+
+*Concepts and Constraints are not supported, its usage is non-trivial substitution.*
+
+### The Data & Interface
+
+As mentioned in [Usage](#usage), the user is provided Code objects by calling the constructor's functions to generate them or find existing matches.
+
+The AST is managed by the library and provided the user via its interface.  
+However, the user may specifiy memory configuration.
+
+Data layout of AST struct:
+
+```cpp
+union {
+    struct
+    {
+        AST*      Attributes;     // Class, Enum, Function, Struct, Typedef, Union, Using, Variable
+        AST*      Specs;          // Function, Operator, Type symbol, Variable
+        union {
+            AST*  ParentType;     // Class, Struct
+            AST*  ReturnType;     // Function, Operator
+            AST*  UnderlyingType; // Enum, Typedef
+            AST*  ValueType;      // Parameter, Variable
+        };
+        AST*      Params;         // Function, Operator, Template
+        union {
+            AST*  ArrExpr;        // Type Symbol
+            AST*  Body;           // Class, Enum, Function, Namespace, Struct, Union
+            AST*  Declaration;    // Friend, Template
+            AST*  Value;          // Parameter, Variable
+        };
+    };
+    StringCached  Content;        // Attributes, Comment, Execution, Include
+    SpecifierT    ArrSpecs[AST::ArrSpecs_Cap]; // Specifiers
+};
+union {
+    AST* Prev;
+    AST* Front; // Used by CodeBody
+    AST* Last;  // Used by CodeParam
+};
+union {
+    AST* Next;
+    AST* Back;  // Used by CodeBody
+};
+AST*              Parent;
+StringCached      Name;
+CodeT             Type;
+ModuleFlag        ModuleFlags;
+union {
+    OperatorT     Op;
+    AccessSpec    ParentAccess;
+    s32           NumEntries;
+};
+```
+
+*`CodeT` is a typedef for `ECode::Type` which has an underlying type of `u32`*  
+*`OperatorT` is a typedef for `EOperator::Type` which has an underlying type of `u32`*  
+*`StringCahced` is a typedef for `String const`, to denote it is an interned string*  
+*`String` is the dynamically allocated string type for the library*
+
+AST widths are setup to be AST_POD_Size.
+The width dictates how much the static array can hold before it must give way to using an allocated array:
+
+```cpp
+constexpr static
+uw ArrSpecs_Cap =
+(
+    AST_POD_Size
+    - sizeof(AST*) * 3
+    - sizeof(StringCached)
+    - sizeof(CodeT)
+    - sizeof(ModuleFlag)
+    - sizeof(s32)
+)
+/ sizeof(SpecifierT) -1; // -1 for 4 extra bytes (Odd num of AST*)
+```
+
+*Ex: If the AST_POD_Size is 128 the capacity of the static array is 20.*
+
+Data Notes:
+
+* The allocator definitions used are exposed to the user incase they want to dictate memory usage
+  * You'll find the memory handling in `init`, `gen_string_allocator`, `get_cached_string`, `make_code`.
+* ASTs are wrapped for the user in a Code struct which is a wrapper for a AST* type.
+* Both AST and Code have member symbols but their data layout is enforced to be POD types.
+* This library treats memory failures as fatal.
+* Cached Strings are stored in their own set of arenas. AST constructors use cached strings for names, and content.
+  * `StringArenas`, `StringCache`, `Allocator_StringArena`, and `Allocator_StringTable` are the associated containers or allocators.
+* Strings used for serialization and file buffers are not contained by those used for cached strings.
+  * They are currently using `GlobalAllocator`, which are tracked array of arenas that grows as needed (adds buckets when one runs out).
+  * Memory within the buckets is not reused, so its inherently wasteful.
+  * I will be augmenting the single arena with a simple slag allocator.
+* Linked lists used children nodes on bodies, and parameters.
+* Its intended to generate the AST in one go and serialize after. The constructors and serializer are designed to be a "one pass, front to back" setup.
+* Allocations can be tuned by defining the folloiwng macros:
+  * `GEN_BUILDER_STR_BUFFER_RESERVE`
+  * `GEN_CODEPOOL_NUM_BLOCKS` : Number of blocks per code pool in the code allocator
+  * `GEN_GLOBAL_BUCKET_SIZE` : Size of each bucket area for the global allocator
+  * `GEN_LEX_ALLOCATOR_SIZE`
+  * `GEN_MAX_COMMENT_LINE_LENGTH` : Longest length a comment can have per line.
+  * `GEN_MAX_NAME_LENGTH` : Max length of any identifier.
+  * `GEN_MAX_UNTYPED_STR_LENGTH` : Max content length for any untyped code.
+  * `GEN_SIZE_PER_STRING_ARENA` : Size per arena used with string caching.
+  * `GEN_TOKEN_FMT_TOKEN_MAP_MEM_SIZE` : token_fmt_va uses local_persit memory of this size for the hashtable.
+
+The following CodeTypes are used which the user may optionally use strong typing with if they enable: `GEN_ENFORCE_STRONG_CODE_TYPES`
+
+* CodeBody : Has support for `for-range` iterating across Code objects.
+* CodeAttributes
+* CodeComment
+* CodeClass
+* CodeConstructor
+* CodeDefine
+* CodeDestructor
+* CodePreprocessCond
+* CodeEnum
+* CodeExec
+* CodeExtern
+* CodeInclude
+* CodeFriend
+* CodeFn
+* CodeModule
+* CodeNamespace
+* CodeOperator
+* CodeOpCast
+* CodeParam : Has support for `for-range` iterating across parameters.
+* CodeSpecifiers : Has support for `for-range` iterating across specifiers.
+* CodeStruct
+* CodeTemplate
+* CodeType
+* CodeTypedef
+* CodeUnion
+* CodeUsing
+* CodeVar
+
+Each Code boy has an associated "filtered AST" with the naming convention: `AST_<CodeName>`
+Unrelated fields of the AST for that node type are omitted and only necessary padding members are defined otherwise.
+Retrieving a raw version of the ast can be done using the `raw()` function defined in each AST.
+
+## There are three sets of interfaces for Code AST generation the library provides
+
+* Upfront
+* Parsing
+* Untyped
+
+### Upfront Construction
+
+All component ASTs must be previously constructed, and provided on creation of the code AST.
+The construction will fail and return Code::Invalid otherwise.
+
+Interface :``
+
+* def_attributes
+  * *This is pre-appended right before the function symbol, or placed after the class or struct keyword for any flavor of attributes used.*
+  * *Its up to the user to use the desired attribute formatting: `[[]]` (standard), `__declspec` (Microsoft), or `__attribute__` (GNU).*
+* def_comment
+* def_class
+* def_constructor
+* def_destructor
+* def_enum
+* def_execution
+  * *This is equivalent to untyped_str, except that its intended for use only in execution scopes.*
+* def_extern_link
+* def_friend
+* def_function
+* def_include
+* def_module
+* def_namespace
+* def_operator
+* def_operator_cast
+* def_param
+* def_params
+* def_specifier
+* def_specifiers
+* def_struct
+* def_template
+* def_type
+* def_typedef
+* def_union
+* def_using
+* def_using_namespace
+* def_variable
+
+Bodies:
+
+* def_body
+* def_class_body
+* def_enum_body
+* def_export_body
+* def_extern_link_body
+* def_function_body
+  * *Use this for operator bodies as well*
+* def_global_body
+* def_namespace_body
+* def_struct_body
+* def_union_body
+
+Usage:
+
+```cpp
+<name> = def_<function type>( ... );
+
+Code <name>
+{
+    ...
+    <name> = def_<function name>( ... );
+}
+
+```
+
+When using the body functions, its recommended to use the args macro to auto determine the number of arguments for the varadic:
+```cpp
+def_global_body( args( ht_entry, array_ht_entry, hashtable ));
+
+// instead of:
+def_global_body( 3, ht_entry, array_ht_entry, hashtable );
+```
+
+If a more incremental approach is desired for the body ASTs, `Code def_body( CodeT type )` can be used to create an empty body.  
+When the members have been populated use: `AST::validate_body` to verify that the members are valid entires for that type.
+
+### Parse construction
+
+A string provided to the API is parsed for the intended language construct.
+
+Interface :
+
+* parse_class
+* parse_constructor
+* parse_destructor
+* parse_enum
+* parse_export_body
+* parse_extern_link
+* parse_friend
+  * Purposefully are only support forward declares with this constructor.
+* parse_function
+* parse_global_body
+* parse_namespace
+* parse_operator
+* parse_operator_cast
+* parse_struct
+* parse_template
+* parse_type
+* parse_typedef
+* parse_union
+* parse_using
+* parse_variable
+
+The lexing and parsing takes shortcuts from whats expected in the standard.
+
+* Numeric literals are not check for validity.
+* The parse API treats any execution scope definitions with no validation and are turned into untyped Code ASTs.
+  * *This includes the assignment of variables.*
+* Attributes ( `[[]]` (standard), `__declspec` (Microsoft), or `__attribute__` (GNU) )
+  * Assumed to *come before specifiers* (`const`, `constexpr`, `extern`, `static`, etc) for a function
+  * Or in the usual spot for class, structs, (*right after the declaration keyword*)
+  * typedefs have attributes with the type (`parse_type`)
+* As a general rule; if its not available from the upfront constructors, its not available in the parsing constructors.
+  * *Upfront constructors are not necessarily used in the parsing constructors, this is just a good metric to know what can be parsed.*
+* Parsing attributes can be extended to support user defined macros by defining `GEN_DEFINE_ATTRIBUTE_TOKENS` (see `gen.hpp` for the formatting)
+
+Usage:
+
+```cpp
+Code <name> = parse_<function name>( string with code );
+
+Code <name> = def_<function name>( ..., parse_<function name>(
+    <string with code>
+));
+
+Code <name> = make_<function name>( ... )
+{
+    <name>->add( parse_<function name>(
+        <string with code>
+    ));
+}
+```
+
+### Untyped constructions
+
+Code ASTs are constructed using unvalidated strings.
+
+Interface :
+
+* token_fmt_va
+* token_fmt
+* untyped_str
+* untyped_fmt
+* untyped_token_fmt
+
+During serialization any untyped Code AST has its string value directly injected inline of whatever context the content existed as an entry within.  
+Even though these are not validated from somewhat correct c/c++ syntax or components, it doesn't mean that Untyped code can be added as any component of a Code AST:
+
+* Untyped code cannot have children, thus there cannot be recursive injection this way.
+* Untyped code can only be a child of a parent of body AST, or for values of an assignment (ex: variable assignment).
+
+These restrictions help prevent abuse of untyped code to some extent.
+
+Usage Conventions:
+
+```cpp
+Code <name> = def_variable( <type>, <name>, untyped_<function name>(
+    <string with code>
+));
+
+Code <name> = untyped_str( code(
+    <some code without "" quotes>
+));
+```
+
+Optionally, `code_str`, and `code_fmt` macros can be used so that the code macro doesn't have to be used:
+```cpp
+Code <name> = code_str( <some code without "" quotes > )
+```
+
+Template metaprogramming in the traditional sense becomes possible with the use of `token_fmt` and parse constructors:
+
+```cpp
+StrC value = txt_StrC("Something");
+
+char const* template_str = txt(
+    Code with <key> to replace with token_values
+    ...
+);
+char const* gen_code_str = token_fmt( "key", value, template_str );
+Code        <name>       = parse_<function name>( gen_code_str );
+```
+
+## Predefined Codes
+
+The following are provided predefined by the library as they are commonly used:
+
+* `access_public`
+* `access_protected`
+* `access_private`
+* `module_global_fragment`
+* `module_private_fragment`
+* `param_varaidc` (Used for varadic definitions)
+* `preprocess_else`
+* `preprocess_endif`
+* `pragma_once`
+* `spec_const`
+* `spec_consteval`
+* `spec_constexpr`
+* `spec_constinit`
+* `spec_extern_linkage` (extern)
+* `spec_final`
+* `spec_global` (global macro)
+* `spec_inline`
+* `spec_internal_linkage` (internal macro)
+* `spec_local_persist` (local_persist macro)
+* `spec_mutable`
+* `spec_override`
+* `spec_ptr`
+* `spec_ref`
+* `spec_register`
+* `spec_rvalue`
+* `spec_static_member` (static)
+* `spec_thread_local`
+* `spec_virtual`
+* `spec_volatile`
+* `spec_type_signed`
+* `spec_type_unsigned`
+* `spec_type_short`
+* `spec_type_long`
+* `t_empty` (Used for varaidc macros)
+* `t_auto`
+* `t_void`
+* `t_int`
+* `t_bool`
+* `t_char`
+* `t_wchar_t`
+* `t_class`
+* `t_typename`
+
+Optionally the following may be defined if `GEN_DEFINE_LIBRARY_CODE_CONSTANTS` is defined
+
+* `t_b32`
+* `t_s8`
+* `t_s16`
+* `t_s32`
+* `t_s64`
+* `t_u8`
+* `t_u16`
+* `t_u32`
+* `t_u64`
+* `t_sw`
+* `t_uw`
+* `t_f32`
+* `t_f64`
+
+## Extent of operator overload validation
+
+The AST and constructors will be able to validate that the arguments provided for the operator type match the expected form:
+
+* If return type must match a parameter
+* If number of parameters is correct
+* If added as a member symbol to a class or struct, that operator matches the requirements for the class (types match up)
+
+The user is responsible for making sure the code types provided are correct
+and have the desired specifiers assigned to them beforehand.
+
+## Code generation and modification
+
+There are three provided file interfaces:
+
+* Builder
+* Editor
+* Scanner
+
+Editor and Scanner are disabled by default, use `GEN_FEATURE_EDITOR` and `GEN_FEATURE_SCANNER` to enable them.
+
+### Builder is a similar object to the jai language's string_builder
+
+* The purpose of it is to generate a file.
+* A file is specified and opened for writing using the open( file_path) ) function.
+* The code is provided via print( code ) function  will be serialized to its buffer.
+* When all serialization is finished, use the write() command to write the buffer to the file.
+
+### Editor is for editing a series of files based on a set of requests provided to it
+
+**Note: Not implemented yet**
+
+* The purpose is to overrite a specific file, it places its contents in a buffer to scan.
+* Requests are populated using the following interface:
+  * add : Add code.
+  * remove : Remove code.
+  * replace: Replace code.
+
+All three have the same parameters with exception to remove which only has SymbolInfo and Policy:
+
+* SymbolInfo:
+  * File      : The file the symbol resides in. Leave null to indicate to search all files. Leave null to indicated all-file search.
+  * Marker    : #define symbol that indicates a location or following signature is valid to manipulate. Leave null to indicate the signature should only be used.
+  * Signature : Use a Code symbol to find a valid location to manipulate, can be further filtered with the marker. Leave null to indicate the marker should only be used.
+
+* Policy : Additional policy info for completing the request (empty for now)
+* Code   : Code to inject if adding, or replace existing code with.
+
+Additionally if `GEN_FEATURE_EDITOR_REFACTOR` is defined, refactor( file_path, specification_path ) wil be made available.  
+Refactor is based of the refactor library and uses its interface.  
+It will on call add a request to the queue to run the refactor script on the file.
+
+### Scanner allows the user to generate Code ASTs by reading files
+
+**Note: Not implemented yet**
+
+* The purpose is to grab definitions to generate metadata or generate new code from these definitions.
+* Requests are populated using the add( SymbolInfo, Policy ) function. The symbol info is the same as the one used for the editor. So is the case with Policy.
+
+The file will only be read from, no writing supported.
+
+One great use case is for example: generating the single-header library for gencpp!
+
+### Additional Info (Editor and Scanner)
+
+When all requests have been populated, call process_requests().  
+It will provide an output of receipt data of the results when it completes.
+
+Files may be added to the Editor and Scanner additionally with add_files( num, files ).  
+This is intended for when you have requests that are for multiple files.
+
+Request queue in both Editor and Scanner are cleared once process_requests completes.

Readme.md

@@ -12,43 +12,12 @@ its not meant to be a black box metaprogramming utility, its meant for the user
 
 * [Notes](#notes)
 * [Usage](#usage)
-* [Building](#notes)
+* [Building](#building)
-* [Outline](#outline)
-* [What is not provided](#what-is-not-provided)
-* [The three constructors ](#there-are-three-sets-of-interfaces-for-code-ast-generation-the-library-provides)
-* [Predefined Codes](#predefined-codes)
-* [Code generation and modification](#code-generation-and-modification)
 
 ## Notes
 
 The project has reached an *alpha* state, all the current functionality works for the test cases but it will most likely break in many other cases.  
-
+The [issues](https://github.com/Ed94/gencpp/issues) marked with v1.0 Feature indicate whats left before the library is considered feature complete.
-The project has no external dependencies beyond:
-
-* `errno.h`
-* `stat.h`
-* `stdarg.h`
-* `stddef.h`
-* `stdio.h`
-* `copyfile.h` (Mac)
-* `types.h`    (Linux)
-* `unistd.h`   (Linux/Mac)
-* `intrin.h`   (Windows)
-* `io.h`       (Windows with gcc)
-* `windows.h`  (Windows)
-
-Dependencies for the project are wrapped within `GENCPP_ROLL_OWN_DEPENDENCIES` (Defining it will disable them).  
-The majority of the dependency's implementation was derived from the [c-zpl library](https://github.com/zpl-c/zpl).
-
-This library was written a subset of C++ where the following are not used at all:
-
-* RAII (Constructors/Destructors), lifetimes are managed using named static or regular functions.
-* Language provide dynamic dispatch, RTTI
-* Object-Oriented Inheritance
-* Exceptions
-
-Polymorphic & Member-functions are used as an ergonomic choice, along with a conserative use of operator overloads.  
-There are only 4 template definitions in the entire library. (`Array<Type>`, `Hashtable<Type>`, `swap<Type>`, and `AST/Code::cast<Type>`)
 
 A `natvis` and `natstepfilter` are provided in the scripts directory.
 
@@ -87,7 +56,7 @@ u32 gen_main()
 ```
 
 The design uses a constructive builder API for the code to generate.  
-The user is given `Code` objects that are used to build up the AST.
+The user is provided `Code` objects that are used to build up the AST.
 
 Example using each construction interface:
 
@@ -154,529 +123,8 @@ struct ArrayHeader
 ```
 
 **Note: The formatting shown here is not how it will look. For your desired formatting its recommended to run a pass through the files with an auto-formatter.**
+*(The library currently uses clang-format for formatting, beaware its pretty slow...)*
 
 ## Building
 
-An example of building is provided within project, singleheader, and test.
+See the [scripts directory](scripts/).
-
-(All generated files go to a corresponding `*/gen` directory)
-
-**Project**
-
-`gen.bootstrap.cpp` generates a segmented version of the library where code is divided into `gen.hpp`, `gen.cpp`, `gen_dep.hpp`, and `gen_dep.cpp`.
-(Similar whats there already, but with the includes injected)
-
-**Singleheader**
-
-`gen.singleheader.cpp` generates a single-header version of the library following the convention shown in popular libraries such as: gb, stb, and zpl.
-
-**Test**
-
-There are two meson build files the one within test is the program's build specification.  
-The other one in the gen directory within test is the metaprogram's build specification.
-
-Both of them build the same source file: `test.cpp`. The only differences are that gen needs a different relative path to the include directories and defines the macro definition: `GEN_TIME`.
-
-This method is setup where all the metaprogram's code are the within the same files as the regular program's code.  
-
-## Outline
-
-### *WHAT IS NOT PROVIDED*
-
-* Exceptions
-* Execution statement validation : Execution expressions are defined using the untyped API.
-  * Lambdas (This naturally means its unsupported)
-* RAII : This needs support for constructors/destructor parsing
-  * Haven't gotten around to yet, only useful (to me) for third-party scanning
-
-Keywords kept from "Modern C++":
-
-* constexpr : Great to store compile-time constants.
-* consteval : Technically fine, need to make sure to execute in moderation.
-* constinit : Better than constexpr at doing its job, however, its only c++ 20.
-* export    : Useful if c++ modules ever come around to actually being usable.
-* import    : ^^
-* module    : ^^
-
-When it comes to expressions:
-
-**There is no support for validating expressions.**  
-Its difficult to parse without enough benefits (At the metaprogramming level).  
-
-When it comes to templates:
-
-Only trivial template support is provided. the intention is for only simple, non-recursive substitution.  
-The parameters of the template are treated like regular parameter AST entries.  
-This means that the typename entry for the parameter AST would be either:
-
-* `class`
-* `typename`
-* A fundamental type, function, or pointer type.
-
-Anything beyond this usage is not supported by parse_template for arguments (at least not intentionally).  
-Use at your own mental peril.
-
-*Concepts and Constraints are not supported, its usage is non-trivial substitution.*
-
-### The Data & Interface
-
-As mentioned in [Usage](#usage), the user is provided Code objects by calling the constructor's functions to generate them or find existing matches.
-
-The AST is managed by the library and provided the user via its interface.  
-However, the user may specifiy memory configuration.
-
-Data layout of AST struct:
-
-```cpp
-union {
-    struct
-    {
-        AST*      Attributes;     // Class, Enum, Function, Struct, Typedef, Union, Using, Variable
-        AST*      Specs;          // Function, Operator, Type symbol, Variable
-        union {
-            AST*  ParentType;     // Class, Struct
-            AST*  ReturnType;     // Function, Operator
-            AST*  UnderlyingType; // Enum, Typedef
-            AST*  ValueType;      // Parameter, Variable
-        };
-        AST*      Params;         // Function, Operator, Template
-        union {
-            AST*  ArrExpr;        // Type Symbol
-            AST*  Body;           // Class, Enum, Function, Namespace, Struct, Union
-            AST*  Declaration;    // Friend, Template
-            AST*  Value;          // Parameter, Variable
-        };
-    };
-    StringCached  Content;        // Attributes, Comment, Execution, Include
-    SpecifierT    ArrSpecs[AST::ArrSpecs_Cap]; // Specifiers
-};
-union {
-    AST* Prev;
-    AST* Front; // Used by CodeBody
-    AST* Last;  // Used by CodeParam
-};
-union {
-    AST* Next;
-    AST* Back;  // Used by CodeBody
-};
-AST*              Parent;
-StringCached      Name;
-CodeT             Type;
-ModuleFlag        ModuleFlags;
-union {
-    OperatorT     Op;
-    AccessSpec    ParentAccess;
-    s32           NumEntries;
-};
-```
-
-*`CodeT` is a typedef for `ECode::Type` which has an underlying type of `u32`*  
-*`OperatorT` is a typedef for `EOperator::Type` which has an underlying type of `u32`*  
-*`StringCahced` is a typedef for `String const`, to denote it is an interned string*  
-*`String` is the dynamically allocated string type for the library*
-
-AST widths are setup to be AST_POD_Size.
-The width dictates how much the static array can hold before it must give way to using an allocated array:
-
-```cpp
-constexpr static
-uw ArrSpecs_Cap =
-(
-    AST_POD_Size
-    - sizeof(AST*) * 3
-    - sizeof(StringCached)
-    - sizeof(CodeT)
-    - sizeof(ModuleFlag)
-    - sizeof(s32)
-)
-/ sizeof(SpecifierT) -1; // -1 for 4 extra bytes (Odd num of AST*)
-```
-
-*Ex: If the AST_POD_Size is 128 the capacity of the static array is 20.*
-
-Data Notes:
-
-* The allocator definitions used are exposed to the user incase they want to dictate memory usage
-  * You'll find the memory handling in `init`, `gen_string_allocator`, `get_cached_string`, `make_code`.
-* ASTs are wrapped for the user in a Code struct which is a wrapper for a AST* type.
-* Both AST and Code have member symbols but their data layout is enforced to be POD types.
-* This library treats memory failures as fatal.
-* Cached Strings are stored in their own set of arenas. AST constructors use cached strings for names, and content.
-  * `StringArenas`, `StringCache`, `Allocator_StringArena`, and `Allocator_StringTable` are the associated containers or allocators.
-* Strings used for serialization and file buffers are not contained by those used for cached strings.
-  * They are currently using `GlobalAllocator`, which are tracked array of arenas that grows as needed (adds buckets when one runs out).
-  * Memory within the buckets is not reused, so its inherently wasteful.
-  * I will be augmenting the single arena with a simple slag allocator.
-* Linked lists used children nodes on bodies, and parameters.
-* Its intended to generate the AST in one go and serialize after. The constructors and serializer are designed to be a "one pass, front to back" setup.
-* Allocations can be tuned by defining the folloiwng macros:
-  * `GEN_GLOBAL_BUCKET_SIZE` : Size of each bucket area for the global allocator
-  * `GEN_CODEPOOL_NUM_BLOCKS` : Number of blocks per code pool in the code allocator
-  * `GEN_SIZE_PER_STRING_ARENA` : Size per arena used with string caching.
-  * `GEN_MAX_COMMENT_LINE_LENGTH` : Longest length a comment can have per line.
-  * `GEN_MAX_NAME_LENGTH` : Max length of any identifier.
-  * `GEN_MAX_UNTYPED_STR_LENGTH` : Max content length for any untyped code.
-  * `GEN_TOKEN_FMT_TOKEN_MAP_MEM_SIZE` : token_fmt_va uses local_persit memory of this size for the hashtable.
-  * `GEN_LEX_ALLOCATOR_SIZE`
-  * `GEN_BUILDER_STR_BUFFER_RESERVE`
-
-Two generic templated containers are used throughout the library:
-
-* `template< class Type> struct Array`
-* `template< class Type> struct HashTable`
-
-Both Code and AST definitions have a `template< class Type> Code/AST cast()`. Its just an alternative way to explicitly cast to each other.
-
-`template< class Type> swap( Type& a, Type& b)` is used over a macro.
-
-Otherwise the library is free of any templates.
-
-The following CodeTypes are used which the user may optionally use strong typing with if they enable: `GEN_ENFORCE_STRONG_CODE_TYPES`
-
-* CodeBody : Has support for `for-range` iterating across Code objects.
-* CodeAttributes
-* CodeComment
-* CodeClass
-* CodeEnum
-* CodeExec
-* CodeExtern
-* CodeInclude
-* CodeFriend
-* CodeFn
-* CodeModule
-* CodeNamespace
-* CodeOperator
-* CodeOpCast
-* CodeParam : Has support for `for-range` iterating across parameters.
-* CodeSpecifiers : Has support for `for-range` iterating across specifiers.
-* CodeStruct
-* CodeTemplate
-* CodeType
-* CodeTypedef
-* CodeUnion
-* CodeUsing
-* CodeUsingNamespace
-* CodeVar
-
-Each Code boy has an associated "filtered AST" with the naming convention: `AST_<CodeName>`
-Unrelated fields of the AST for that node type are omitted and only necessary padding members are defined otherwise.
-Retrieving a raw version of the ast can be done using the `raw()` function defined in each AST.
-
-## There are three sets of interfaces for Code AST generation the library provides
-
-* Upfront
-* Parsing
-* Untyped
-
-### Upfront Construction
-
-All component ASTs must be previously constructed, and provided on creation of the code AST.
-The construction will fail and return Code::Invalid otherwise.
-
-Interface :``
-
-* def_attributes
-  * *This is pre-appended right before the function symbol, or placed after the class or struct keyword for any flavor of attributes used.*
-  * *Its up to the user to use the desired attribute formatting: `[[]]` (standard), `__declspec` (Microsoft), or `__attribute__` (GNU).*
-* def_comment
-* def_class
-* def_enum
-* def_execution
-  * *This is equivalent to untyped_str, except that its intended for use only in execution scopes.*
-* def_extern_link
-* def_friend
-* def_function
-* def_include
-* def_module
-* def_namespace
-* def_operator
-* def_operator_cast
-* def_param
-* def_params
-* def_specifier
-* def_specifiers
-* def_struct
-* def_template
-* def_type
-* def_typedef
-* def_union
-* def_using
-* def_using_namespace
-* def_variable
-
-Bodies:
-
-* def_body
-* def_class_body
-* def_enum_body
-* def_export_body
-* def_extern_link_body
-* def_function_body
-  * *Use this for operator bodies as well*
-* def_global_body
-* def_namespace_body
-* def_struct_body
-* def_union_body
-
-Usage:
-
-```cpp
-<name> = def_<function type>( ... );
-
-Code <name>
-{
-    ...
-    <name> = def_<function name>( ... );
-}
-
-```
-
-When using the body functions, its recommended to use the args macro to auto determine the number of arguments for the varadic:
-```cpp
-def_global_body( args( ht_entry, array_ht_entry, hashtable ));
-
-// instead of:
-def_global_body( 3, ht_entry, array_ht_entry, hashtable );
-```
-
-If a more incremental approach is desired for the body ASTs, `Code def_body( CodeT type )` can be used to create an empty body.  
-When the members have been populated use: `AST::validate_body` to verify that the members are valid entires for that type.
-
-### Parse construction
-
-A string provided to the API is parsed for the intended language construct.
-
-Interface :
-
-* parse_class
-* parse_enum
-* parse_export_body
-* parse_extern_link
-* parse_friend
-  * Purposefully are only support forward declares with this constructor.
-* parse_function
-* parse_global_body
-* parse_namespace
-* parse_operator
-* parse_operator_cast
-* parse_struct
-* parse_template
-* parse_type
-* parse_typedef
-* parse_union
-* parse_using
-* parse_variable
-
-The lexing and parsing takes shortcuts from whats expected in the standard.
-
-* Numeric literals are not check for validity.
-* The parse API treats any execution scope definitions with no validation and are turned into untyped Code ASTs.
-  * *This includes the assignment of variables.*
-* Attributes ( `[[]]` (standard), `__declspec` (Microsoft), or `__attribute__` (GNU) )
-  * Assumed to *come before specifiers* (`const`, `constexpr`, `extern`, `static`, etc) for a function
-  * Or in the usual spot for class, structs, (*right after the declaration keyword*)
-  * typedefs have attributes with the type (`parse_type`)
-* As a general rule; if its not available from the upfront constructors, its not available in the parsing constructors.
-  * *Upfront constructors are not necessarily used in the parsing constructors, this is just a good metric to know what can be parsed.*
-* Parsing attributes can be extended to support user defined macros by defining `GEN_Define_Attribute_Tokens` (see `gen.hpp` for the formatting)
-
-Usage:
-
-```cpp
-Code <name> = parse_<function name>( string with code );
-
-Code <name> = def_<function name>( ..., parse_<function name>(
-    <string with code>
-));
-
-Code <name> = make_<function name>( ... )
-{
-    <name>->add( parse_<function name>(
-        <string with code>
-    ));
-}
-```
-
-### Untyped constructions
-
-Code ASTs are constructed using unvalidated strings.
-
-Interface :
-
-* token_fmt_va
-* token_fmt
-* untyped_str
-* untyped_fmt
-* untyped_token_fmt
-
-During serialization any untyped Code AST has its string value directly injected inline of whatever context the content existed as an entry within.  
-Even though these are not validated from somewhat correct c/c++ syntax or components, it doesn't mean that Untyped code can be added as any component of a Code AST:
-
-* Untyped code cannot have children, thus there cannot be recursive injection this way.
-* Untyped code can only be a child of a parent of body AST, or for values of an assignment (ex: variable assignment).
-
-These restrictions help prevent abuse of untyped code to some extent.
-
-Usage Conventions:
-
-```cpp
-Code <name> = def_variable( <type>, <name>, untyped_<function name>(
-    <string with code>
-));
-
-Code <name> = untyped_str( code(
-    <some code without "" quotes>
-));
-```
-
-Optionally, `code_str`, and `code_fmt` macros can be used so that the code macro doesn't have to be used:
-```cpp
-Code <name> = code_str( <some code without "" quotes > )
-```
-
-Template metaprogramming in the traditional sense becomes possible with the use of `token_fmt` and parse constructors:
-
-```cpp
-StrC value = txt_StrC("Something");
-
-char const* template_str = txt(
-    Code with <key> to replace with token_values
-    ...
-);
-char const* gen_code_str = token_fmt( "key", value, template_str );
-Code        <name>       = parse_<function name>( gen_code_str );
-```
-
-## Predefined Codes
-
-The following are provided predefined by the library as they are commonly used:
-
-* `access_public`
-* `access_protected`
-* `access_private`
-* `module_global_fragment`
-* `module_private_fragment`
-* `param_varaidc` (Used for varadic definitions)
-* `pragma_once`
-* `spec_const`
-* `spec_consteval`
-* `spec_constexpr`
-* `spec_constinit`
-* `spec_extern_linkage` (extern)
-* `spec_final`
-* `spec_global` (global macro)
-* `spec_inline`
-* `spec_internal_linkage` (internal macro)
-* `spec_local_persist` (local_persist macro)
-* `spec_mutable`
-* `spec_override`
-* `spec_ptr`
-* `spec_ref`
-* `spec_register`
-* `spec_rvalue`
-* `spec_static_member` (static)
-* `spec_thread_local`
-* `spec_virtual`
-* `spec_volatile`
-* `spec_type_signed`
-* `spec_type_unsigned`
-* `spec_type_short`
-* `spec_type_long`
-* `t_empty` (Used for varaidc macros)
-* `t_auto`
-* `t_void`
-* `t_int`
-* `t_bool`
-* `t_char`
-* `t_wchar_t`
-* `t_class`
-* `t_typename`
-
-Optionally the following may be defined if `GEN_DEFINE_LIBRARY_CODE_CONSTANTS` is defined
-
-* `t_b32`
-* `t_s8`
-* `t_s16`
-* `t_s32`
-* `t_s64`
-* `t_u8`
-* `t_u16`
-* `t_u32`
-* `t_u64`
-* `t_sw`
-* `t_uw`
-* `t_f32`
-* `t_f64`
-
-## Extent of operator overload validation
-
-The AST and constructors will be able to validate that the arguments provided for the operator type match the expected form:
-
-* If return type must match a parameter
-* If number of parameters is correct
-* If added as a member symbol to a class or struct, that operator matches the requirements for the class (types match up)
-
-The user is responsible for making sure the code types provided are correct
-and have the desired specifiers assigned to them beforehand.
-
-## Code generation and modification
-
-There are three provided file interfaces:
-
-* Builder
-* Editor
-* Scanner
-
-Editor and Scanner are disabled by default, use `GEN_FEATURE_EDITOR` and `GEN_FEATURE_SCANNER` to enable them.
-
-### Builder is a similar object to the jai language's string_builder
-
-* The purpose of it is to generate a file.
-* A file is specified and opened for writing using the open( file_path) ) function.
-* The code is provided via print( code ) function  will be serialized to its buffer.
-* When all serialization is finished, use the write() command to write the buffer to the file.
-
-### Editor is for editing a series of files based on a set of requests provided to it
-
-**Note: Not implemented yet**
-
-* The purpose is to overrite a specific file, it places its contents in a buffer to scan.
-* Requests are populated using the following interface:
-  * add : Add code.
-  * remove : Remove code.
-  * replace: Replace code.
-
-All three have the same parameters with exception to remove which only has SymbolInfo and Policy:
-
-* SymbolInfo:
-  * File      : The file the symbol resides in. Leave null to indicate to search all files. Leave null to indicated all-file search.
-  * Marker    : #define symbol that indicates a location or following signature is valid to manipulate. Leave null to indicate the signature should only be used.
-  * Signature : Use a Code symbol to find a valid location to manipulate, can be further filtered with the marker. Leave null to indicate the marker should only be used.
-
-* Policy : Additional policy info for completing the request (empty for now)
-* Code   : Code to inject if adding, or replace existing code with.
-
-Additionally if `GEN_FEATURE_EDITOR_REFACTOR` is defined, refactor( file_path, specification_path ) wil be made available.  
-Refactor is based of the refactor library and uses its interface.  
-It will on call add a request to the queue to run the refactor script on the file.
-
-### Scanner allows the user to generate Code ASTs by reading files
-
-**Note: Not implemented yet**
-
-* The purpose is to grab definitions to generate metadata or generate new code from these definitions.
-* Requests are populated using the add( SymbolInfo, Policy ) function. The symbol info is the same as the one used for the editor. So is the case with Policy.
-
-The file will only be read from, no writing supported.
-
-One great use case is for example: generating the single-header library for gencpp!
-
-### Additional Info (Editor and Scanner)
-
-When all requests have been populated, call process_requests().  
-It will provide an output of receipt data of the results when it completes.
-
-Files may be added to the Editor and Scanner additionally with add_files( num, files ).  
-This is intended for when you have requests that are for multiple files.
-
-Request queue in both Editor and Scanner are cleared once process_requests completes.

gencpp.vcxproj

@@ -112,12 +112,18 @@
   </ItemGroup>
   <ItemGroup>
     <ClInclude Include="project\components\gen.data_structures.hpp" />
+    <ClInclude Include="project\components\gen.ecode.hpp" />
+    <ClInclude Include="project\components\gen.eoperator.hpp" />
+    <ClInclude Include="project\components\gen.especifier.hpp" />
+    <ClInclude Include="project\components\gen.header_end.hpp" />
+    <ClInclude Include="project\components\gen.header_start.hpp" />
     <ClInclude Include="project\components\gen.interface.hpp" />
     <ClInclude Include="project\components\gen.types.hpp" />
     <ClInclude Include="project\dependencies\gen.basic_types.hpp" />
     <ClInclude Include="project\dependencies\gen.containers.hpp" />
-    <ClInclude Include="project\dependencies\gen.dep.hpp" />
+    <ClInclude Include="project\dependencies\gen.debug.hpp" />
     <ClInclude Include="project\dependencies\gen.file_handling.hpp" />
+    <ClInclude Include="project\dependencies\gen.hashing.hpp" />
     <ClInclude Include="project\dependencies\gen.header_start.hpp" />
     <ClInclude Include="project\dependencies\gen.macros.hpp" />
     <ClInclude Include="project\dependencies\gen.memory.hpp" />
@@ -125,25 +131,17 @@
     <ClInclude Include="project\dependencies\gen.printing.hpp" />
     <ClInclude Include="project\dependencies\gen.string.hpp" />
     <ClInclude Include="project\dependencies\gen.string_ops.hpp" />
+    <ClInclude Include="project\dependencies\gen.timing.hpp" />
     <ClInclude Include="project\filesystem\gen.builder.hpp" />
     <ClInclude Include="project\filesystem\gen.editor.hpp" />
     <ClInclude Include="project\filesystem\gen.scanner.hpp" />
     <ClInclude Include="project\gen.dep.hpp" />
-    <ClInclude Include="project\gen.editor.hpp" />
     <ClInclude Include="project\gen.hpp" />
-    <ClInclude Include="project\gen.pop_ignores.inline.hpp" />
-    <ClInclude Include="project\gen.push_ignores.inline.hpp" />
-    <ClInclude Include="project\gen.scanner.hpp" />
-    <ClInclude Include="project\gen.undef.macros.hpp" />
     <ClInclude Include="project\helpers\gen.pop_ignores.inline.hpp" />
     <ClInclude Include="project\helpers\gen.push_ignores.inline.hpp" />
     <ClInclude Include="project\helpers\gen.undef.macros.hpp" />
+    <ClInclude Include="singleheader\components\gen.header_start.hpp" />
     <ClInclude Include="test\DummyInclude.hpp" />
-    <ClInclude Include="test\gen\array.Upfront.gen.hpp" />
-    <ClInclude Include="test\gen\buffer.Upfront.gen.hpp" />
-    <ClInclude Include="test\gen\hashtable.Upfront.gen.hpp" />
-    <ClInclude Include="test\gen\ring.Upfront.gen.hpp" />
-    <ClInclude Include="test\gen\sanity.Upfront.gen.hpp" />
     <ClInclude Include="test\Parsed\Buffer.Parsed.hpp" />
     <ClInclude Include="test\Parsed\HashTable.Parsed.hpp" />
     <ClInclude Include="test\Parsed\Ring.Parsed.hpp" />
@@ -160,18 +158,28 @@
   </ItemGroup>
   <ItemGroup>
     <ClCompile Include="project\components\gen.ast.cpp" />
+    <ClCompile Include="project\components\gen.ast_case_macros.cpp" />
+    <ClCompile Include="project\components\gen.data.cpp" />
+    <ClCompile Include="project\components\gen.etoktype.cpp" />
+    <ClCompile Include="project\components\gen.impl_start.cpp" />
+    <ClCompile Include="project\components\gen.interface.cpp" />
     <ClCompile Include="project\components\gen.interface.parsing.cpp" />
     <ClCompile Include="project\components\gen.interface.upfront.cpp" />
-    <ClCompile Include="project\dependencies\gen.dep.cpp" />
+    <ClCompile Include="project\components\gen.untyped.cpp" />
+    <ClCompile Include="project\dependencies\gen.debug.cpp" />
     <ClCompile Include="project\dependencies\gen.file_handling.cpp" />
+    <ClCompile Include="project\dependencies\gen.hashing.cpp" />
+    <ClCompile Include="project\dependencies\gen.impl_start.cpp" />
     <ClCompile Include="project\dependencies\gen.memory.cpp" />
     <ClCompile Include="project\dependencies\gen.parsing.cpp" />
     <ClCompile Include="project\dependencies\gen.printing.cpp" />
+    <ClCompile Include="project\dependencies\gen.string.cpp" />
+    <ClCompile Include="project\dependencies\gen.string_ops.cpp" />
+    <ClCompile Include="project\dependencies\gen.timing.cpp" />
+    <ClCompile Include="project\gen.bootstrap.cpp" />
     <ClCompile Include="project\gen.cpp" />
     <ClCompile Include="project\gen.dep.cpp" />
-    <ClCompile Include="singleheader\gen\gen.singleheader.cpp" />
+    <ClCompile Include="singleheader\gen.singleheader.cpp" />
-    <ClCompile Include="test\gen\build\meson-private\sanitycheckc.c" />
-    <ClCompile Include="test\gen\build\meson-private\sanitycheckcpp.cc" />
     <ClCompile Include="test\parsing.cpp" />
     <ClCompile Include="test\sanity.cpp" />
     <ClCompile Include="test\SOA.cpp" />
@@ -184,6 +192,23 @@
     <Natvis Include=".vscode\gencpp.natvis" />
     <Natvis Include="scripts\gencpp.natvis" />
   </ItemGroup>
+  <ItemGroup>
+    <Content Include="project\components\AttributeTokens.csv" />
+    <Content Include="project\components\ECode.csv" />
+    <Content Include="project\components\EOperator.csv" />
+    <Content Include="project\components\ESpecifier.csv" />
+    <Content Include="project\components\ETokType.csv" />
+    <Content Include="project\meson.build" />
+    <Content Include="scripts\.clang-format" />
+    <Content Include="scripts\bootstrap.ci.ps1" />
+    <Content Include="scripts\bootstrap.ps1" />
+    <Content Include="scripts\msvc\build_msvc.ps1" />
+    <Content Include="scripts\msvc\devshell.ps1" />
+    <Content Include="scripts\refactor.ps1" />
+    <Content Include="scripts\singleheader.ci.ps1" />
+    <Content Include="scripts\singleheader.ps1" />
+    <Content Include="singleheader\meson.build" />
+  </ItemGroup>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
   <ImportGroup Label="ExtensionTargets">
   </ImportGroup>

project/Readme.md

@@ -1,47 +1,43 @@
 # Documentation
 
-The core library is contained within `gen.hpp` and `gen.cpp`.  
-Things related to the editor and scanner are in their own respective files. (Ex: `gen.scanner.<hpp/cpp>` )
-
-Dependencies are within `gen.dep.<hpp/cpp>`
-
 The library is fragmented into a series of headers and sources files meant to be scanned in and then generated to a tailored format for the target
 `gen` files.
 
+The principal (user) files are `gen.hpp` and `gen.cpp`.  
+They contain includes for its various components: `components/<component_name>.<hpp/cpp>`
+
+Dependencies are bundled into `gen.dep.<hpp/cpp>`.  
+Just like the `gen.<hpp/cpp>` they include their components: `dependencies/<dependency_name>.<hpp/cpp>`
+
+The fle processors are in their own respective files. (Ex: `file_processors/<file_processor>.<hpp/cpp>` )  
+They directly include `depedencies/file_handling.<hpp/cpp>` as the core library does not include file processing by defualt.
+
+**TODO : Right now the library is not finished structurally, as such the first self-hosting iteration is still WIP**  
 Both libraries use *pre-generated* (self-hosting I guess) version of the library to then generate the latest version of itself.  
-(sort of a verification that the generated version is equivalent)
+(sort of a verification that the generated version is equivalent).
 
 The default `gen.bootstrap.cpp` located in the project folder is meant to be produce a standard segmented library, where the components of the library  
-have relatively dedicated header and source files. With dependencies included at the top of the file and each header starting with a pragma once.  
+have relatively dedicated header and source files. Dependencies included at the top of the file and each header starting with a pragma once.  
-This will overwrite the existing library implementation in the immediate directory.
+The output will be in the `project/gen` directory (if the directory does not exist, it will create it).
 
 Use those to get a general idea of how to make your own tailored version.
 
-If the naming convention is undesired, the `gencpp.refactor` script can be used with the [refactor]()
-
 Feature Macros:
 
+* `GEN_DEFINE_ATTRIBUTE_TOKENS` : Allows user to define their own attribute macros for use in parsing. 
+  * This is auto-generated if using the bootstrap or single-header generation
+  * *Note: The user will use the `AttributeTokens.csv` when the library is fully self-hosting.*
+* `GEN_DEFINE_LIBRARY_CORE_CONSTANTS` : Optional typename codes as they are non-standard to C/C++ and not necessary to library usage
 * `GEN_DONT_USE_NAMESPACE` : By default, the library is wrapped in a `gen` namespace, this will disable that expose it to the global scope.
 * `GEN_DONT_ENFORCE_GEN_TIME_GUARD` : By default, the library ( gen.hpp/ gen.cpp ) expects the macro `GEN_TIME` to be defined, this disables that.
-* `GEN_ROLL_OWN_DEPENDENCIES` : Optional override so that user may define the dependencies themselves.
-* `GEN_DEFINE_LIBRARY_CORE_CONSTANTS` : Optional typename codes as they are non-standard to C/C++ and not necessary to library usage
 * `GEN_ENFORCE_STRONG_CODE_TYPES` : Enforces casts to filtered code types.
 * `GEN_EXPOSE_BACKEND` : Will expose symbols meant for internal use only.
-* `GEN_Define_Attribute_Tokens` : Allows user to define their own attribute macros for use in parsing.
+* `GEN_ROLL_OWN_DEPENDENCIES` : Optional override so that user may define the dependencies themselves.
-
-`GEN_USE_RECURSIVE_AST_DUPLICATION` is available but its not well tested and should not need to be used.  
-If constructing ASTs properly. There should be no modification of ASTs, and thus this would never become an issue.  
-(I will probably remove down the line...)
 
 ## On multi-threading
 
 Currently unsupported. The following changes would have to be made:
 
-* Setup static data access with fences if more than one thread will generate ASTs ( or keep a different set for each thread)
-* Make sure local persistent data of functions are also thread local.
-* The builder should be done on a per-thread basis.
-* Due to the design of the editor and scanner, it will most likely be best to make each file a job to process request entries on. Receipts should have an an array to store per thread. They can be combined to the final receipts array when all files have been processed.
-
 ## Extending the library
 
 This library is relatively very small, and can be extended without much hassle.

project/components/ast.cpp

@@ -17,20 +17,9 @@ AST* AST::duplicate()
 
 String AST::to_string()
 {
-#	define ProcessModuleFlags()                                      \
-	if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))  \
-		result.append( "export " );                                  \
-																     \
-	if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Import ))  \
-		result.append( "import " );                                  \
-
 	local_persist thread_local
 	char SerializationLevel = 0;
 
-#if defined(GEN_BENCHMARK) && defined(GEN_BENCHMARK_SERIALIZATION)
-	u64 time_start = time_rel_ms();
-#endif
-
 	// TODO : Need to refactor so that intermeidate strings are freed conviently.
 	String result = String::make( GlobalAllocator, "" );
 
@@ -82,7 +71,8 @@ String AST::to_string()
 
 		case Class:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			if ( Attributes || ParentType )
 			{
@@ -129,7 +119,8 @@ String AST::to_string()
 
 		case Class_Fwd:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			if ( Attributes )
 				result.append_fmt( "class %s %s;", Attributes->to_string(), Name );
@@ -140,7 +131,8 @@ String AST::to_string()
 
 		case Enum:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			if ( Attributes || UnderlyingType )
 			{
@@ -170,7 +162,8 @@ String AST::to_string()
 
 		case Enum_Fwd:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			if ( Attributes )
 				result.append_fmt( "%s ", Attributes->to_string() );
@@ -181,7 +174,8 @@ String AST::to_string()
 
 		case Enum_Class:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			if ( Attributes || UnderlyingType )
 			{
@@ -219,7 +213,8 @@ String AST::to_string()
 
 		case Enum_Class_Fwd:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			result.append( "enum class " );
 
@@ -262,7 +257,8 @@ String AST::to_string()
 
 		case Function:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			if ( Attributes )
 				result.append_fmt( "%s ", Attributes->to_string() );
@@ -301,7 +297,8 @@ String AST::to_string()
 
 		case Function_Fwd:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			if ( Attributes )
 				result.append_fmt( "%s ", Attributes->to_string() );
@@ -347,7 +344,8 @@ String AST::to_string()
 			break;
 
 		case Namespace:
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			result.append_fmt( "namespace %s\n{\n%s}"
 				, Name
@@ -358,7 +356,8 @@ String AST::to_string()
 		case Operator:
 		case Operator_Member:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			if ( Attributes )
 				result.append_fmt( "%s ", Attributes->to_string() );
@@ -395,7 +394,8 @@ String AST::to_string()
 		case Operator_Fwd:
 		case Operator_Member_Fwd:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			if ( Attributes )
 				result.append_fmt( "%s ", Attributes->to_string() );
@@ -513,7 +513,8 @@ String AST::to_string()
 
 		case Struct:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			if ( Name == nullptr)
 			{
@@ -566,7 +567,8 @@ String AST::to_string()
 
 		case Struct_Fwd:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			if ( Attributes )
 				result.append_fmt( "struct %s %s;", Attributes->to_string(), Name );
@@ -577,7 +579,8 @@ String AST::to_string()
 
 		case Template:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			result.append_fmt( "template< %s >\n%s", Params->to_string(), Declaration->to_string() );
 		}
@@ -585,7 +588,8 @@ String AST::to_string()
 
 		case Typedef:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			result.append( "typedef ");
 
@@ -624,7 +628,8 @@ String AST::to_string()
 
 		case Union:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			result.append( "union " );
 
@@ -650,7 +655,8 @@ String AST::to_string()
 
 		case Using:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			if ( Attributes )
 				result.append_fmt( "%s ", Attributes->to_string() );
@@ -675,7 +681,8 @@ String AST::to_string()
 
 		case Variable:
 		{
-			ProcessModuleFlags();
+			if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
+				result.append( "export " );
 
 			if ( Attributes || Specs )
 			{
@@ -726,11 +733,7 @@ String AST::to_string()
 		break;
 	}
 
-#if defined(GEN_BENCHMARK) && defined(GEN_BENCHMARK_SERIALIZATION)
-	log_fmt("AST::to_string() time taken: %llu for: %s\n", time_rel_ms() - time_start, result );
-#endif
 	return result;
-#undef ProcessModuleFlags
 }
 
 bool AST::is_equal( AST* other )
@@ -760,19 +763,19 @@ bool AST::validate_body()
 {
 	using namespace ECode;
 
-#define CheckEntries( Unallowed_Types )                                                                \
+#define CheckEntries( Unallowed_Types )                                                               \
-	do                                                                                                 \
+	do                                                                                                \
-	{                                                                                                  \
+	{                                                                                                 \
-		for ( Code entry : cast<CodeBody>() )                                                          \
+		for ( Code entry : cast<CodeBody>() )                                                         \
-		{                                                                                              \
+		{                                                                                             \
-			switch ( entry->Type )                                                                     \
+			switch ( entry->Type )                                                                    \
-			{                                                                                          \
+			{                                                                                         \
-				Unallowed_Types                                                                        \
+				Unallowed_Types                                                                       \
-					log_failure( "AST::validate_body: Invalid entry in body %s", entry.debug_str() );  \
+					log_failure( "AST::validate_body: Invalid entry in body %s", entry.debug_str() ); \
-					return false;                                                                      \
+					return false;                                                                     \
-			}                                                                                          \
+			}                                                                                         \
-		}                                                                                              \
+		}                                                                                             \
-	}                                                                                                  \
+	}                                                                                                 \
 	while (0);
 
 	switch ( Type )
@@ -825,6 +828,8 @@ bool AST::validate_body()
 	}
 
 	return false;
+
+#undef CheckEntries
 }
 
 #pragma endregion AST

project/components/especifier.hpp

@@ -101,5 +101,4 @@ namespace ESpecifier
 
 #	undef Define_Specifiers
 }
-
 using SpecifierT = ESpecifier::Type;

project/components/etoktype.cpp

@@ -9,8 +9,8 @@ namespace Parser
 	Attributes_Start is only used to indicate the start of the user_defined attribute list.
 */
 
-#ifndef GEN_Define_Attribute_Tokens
+#ifndef GEN_DEFINE_ATTRIBUTE_TOKENS
-#	define GEN_Define_Attribute_Tokens         \
+#	define GEN_DEFINE_ATTRIBUTE_TOKENS         \
 	Entry( API_Export, "GEN_API_Export_Code" ) \
 	Entry( API_Import, "GEN_API_Import_Code" )
 #endif
@@ -97,7 +97,7 @@ namespace Parser
 		{
 		#	define Entry( Name_, Str_ ) Name_,
 			Define_TokType
-			GEN_Define_Attribute_Tokens
+			GEN_DEFINE_ATTRIBUTE_TOKENS
 		#	undef Entry
 			NumTokens,
 		};
@@ -110,7 +110,7 @@ namespace Parser
 			{
 			#	define Entry( Name_, Str_ ) { sizeof(Str_), Str_ },
 				Define_TokType
-				GEN_Define_Attribute_Tokens
+				GEN_DEFINE_ATTRIBUTE_TOKENS
 			#	undef Entry
 			};
 
@@ -137,7 +137,7 @@ namespace Parser
 			{
 			#	define Entry( Name_, Str_ ) Str_,
 				Define_TokType
-				GEN_Define_Attribute_Tokens
+				GEN_DEFINE_ATTRIBUTE_TOKENS
 			#	undef Entry
 			};
 

project/components/interface.cpp

@@ -105,7 +105,7 @@ void define_constants()
 #endif
 #	undef def_constant_code_type
 
-	t_empty = (CodeType) make_code();
+	t_empty          = (CodeType) make_code();
 	t_empty->Type    = ECode::Typename;
 	t_empty->Name    = get_cached_string( txt_StrC("") );
 	t_empty.set_global();
@@ -127,12 +127,12 @@ void define_constants()
 	access_private->Name = get_cached_string( txt_StrC("private:") );
 	access_private.set_global();
 
-	access_protected = make_code();
+	access_protected       = make_code();
 	access_protected->Type = ECode::Access_Protected;
 	access_protected->Name = get_cached_string( txt_StrC("protected:") );
 	access_protected.set_global();
 
-	access_public = make_code();
+	access_public       = make_code();
 	access_public->Type = ECode::Access_Public;
 	access_public->Name = get_cached_string( txt_StrC("public:") );
 	access_public.set_global();

project/components/types.hpp

@@ -68,7 +68,7 @@ ModuleFlag operator|( ModuleFlag A, ModuleFlag B)
 
 	Override these to change the attribute to your own unique identifier convention.
 
-	The tokenizer identifies attribute defines with the GEN_Define_Attribute_Tokens macros.
+	The tokenizer identifies attribute defines with the GEN_DEFINE_ATTRIBUTE_TOKENS macros.
 	See the example below and the Define_TokType macro used in gen.cpp to know the format.
 	While the library can parse raw attributes, most projects use defines to wrap them for compiler
 	platform indendence. The token define allows support for them without having to modify the library.

project/components/untyped.cpp

@@ -12,8 +12,7 @@ sw token_fmt_va( char* buf, uw buf_size, s32 num_tokens, va_list va )
 		char tok_map_mem[ TokenFmt_TokenMap_MemSize ];
 
 		tok_map_arena = Arena::init_from_memory( tok_map_mem, sizeof(tok_map_mem) );
-
+		tok_map       = HashTable<StrC>::init( tok_map_arena );
-		tok_map = HashTable<StrC>::init( tok_map_arena );
 
 		s32 left = num_tokens - 1;
 

project/dependencies/printing.hpp

@@ -10,12 +10,15 @@ struct FileInfo;
 char* str_fmt_buf       ( char const* fmt, ... );
 char* str_fmt_buf_va    ( char const* fmt, va_list va );
 sw    str_fmt_va        ( char* str, sw n, char const* fmt, va_list va );
-sw    str_fmt_file       ( FileInfo* f, char const* fmt, ... );
-sw    str_fmt_file_va    ( FileInfo* f, char const* fmt, va_list va );
 sw    str_fmt_out_va    ( char const* fmt, va_list va );
 sw    str_fmt_out_err   ( char const* fmt, ... );
 sw    str_fmt_out_err_va( char const* fmt, va_list va );
 
+// TODO : Move these to file handling.
+
+sw    str_fmt_file      ( FileInfo* f, char const* fmt, ... );
+sw    str_fmt_file_va   ( FileInfo* f, char const* fmt, va_list va )
+
 constexpr
 char const* Msg_Invalid_Value = "INVALID VALUE PROVIDED";
 

project/gen.bootstrap.cpp

@@ -2,8 +2,8 @@
 #define GEN_ENFORCE_STRONG_CODE_TYPES
 #define GEN_EXPOSE_BACKEND
 #include "gen.cpp"
-#include "filesystem/gen.scanner.hpp"
+#include "filesystem/scanner.hpp"
-#include "helpers/gen.helper.hpp"
+#include "helpers/helper.hpp"
 
 using namespace gen;
 
@@ -33,25 +33,27 @@ int gen_main()
 {
 	gen::init();
 
-	Code push_ignores = scan_file( "helpers/gen.push_ignores.inline.hpp" );
+	Code push_ignores = scan_file( "helpers/push_ignores.inline.hpp" );
-	Code pop_ignores  = scan_file( "helpers/gen.pop_ignores.inline.hpp" );
+	Code pop_ignores  = scan_file( "helpers/pop_ignores.inline.hpp" );
 
 	// gen_dep.hpp
 	{
-		Code header_start  = scan_file( "dependencies/gen.header_start.hpp" );
+		Code header_start  = scan_file( "dependencies/header_start.hpp" );
 		Code nspace_macro  = untyped_str( namespace_by_default ? nspace_default : nspace_non_default );
-		Code macros 	   = scan_file( "dependencies/gen.macros.hpp" );
+		Code macros 	   = scan_file( "dependencies/macros.hpp" );
-		Code basic_types   = scan_file( "dependencies/gen.basic_types.hpp" );
+		Code basic_types   = scan_file( "dependencies/basic_types.hpp" );
-		Code debug         = scan_file( "dependencies/gen.debug.hpp" );
+		Code debug         = scan_file( "dependencies/debug.hpp" );
-		Code memory	       = scan_file( "dependencies/gen.memory.hpp" );
+		Code memory	       = scan_file( "dependencies/memory.hpp" );
-		Code string_ops    = scan_file( "dependencies/gen.string_ops.hpp" );
+		Code string_ops    = scan_file( "dependencies/string_ops.hpp" );
-		Code printing      = scan_file( "dependencies/gen.printing.hpp" );
+		Code printing      = scan_file( "dependencies/printing.hpp" );
-		Code containers    = scan_file( "dependencies/gen.containers.hpp" );
+		Code containers    = scan_file( "dependencies/containers.hpp" );
-		Code hashing 	   = scan_file( "dependencies/gen.hashing.hpp" );
+		Code hashing 	   = scan_file( "dependencies/hashing.hpp" );
-		Code string        = scan_file( "dependencies/gen.string.hpp" );
+		Code string        = scan_file( "dependencies/string.hpp" );
-		Code file_handling = scan_file( "dependencies/gen.file_handling.hpp" );
+		Code parsing       = scan_file( "dependencies/parsing.hpp" );
-		Code parsing       = scan_file( "dependencies/gen.parsing.hpp" );
+		Code timing        = scan_file( "dependencies/timing.hpp" );
-		Code timing        = scan_file( "dependencies/gen.timing.hpp" );
+
+		// TOOD : Make this optional
+		Code file_handling = scan_file( "dependencies/file_handling.hpp" );
 
 		Builder
 		deps_header;
@@ -82,15 +84,15 @@ int gen_main()
 	// gen_dep.cpp
 	{
 		CodeInclude header     = def_include( txt_StrC("gen_dep.hpp") );
-		Code        impl_start = scan_file( "dependencies/gen.impl_start.cpp" );
+		Code        impl_start = scan_file( "dependencies/impl_start.cpp" );
-		Code 	    debug      = scan_file( "dependencies/gen.debug.cpp" );
+		Code 	    debug      = scan_file( "dependencies/debug.cpp" );
-		Code 	    string_ops = scan_file( "dependencies/gen.string_ops.cpp" );
+		Code 	    string_ops = scan_file( "dependencies/string_ops.cpp" );
-		Code 	    printing   = scan_file( "dependencies/gen.printing.cpp" );
+		Code 	    printing   = scan_file( "dependencies/printing.cpp" );
-		Code 	    memory     = scan_file( "dependencies/gen.memory.cpp" );
+		Code 	    memory     = scan_file( "dependencies/memory.cpp" );
-		Code        parsing    = scan_file( "dependencies/gen.parsing.cpp" );
+		Code        parsing    = scan_file( "dependencies/parsing.cpp" );
-		Code        hashing    = scan_file( "dependencies/gen.hashing.cpp" );
+		Code        hashing    = scan_file( "dependencies/hashing.cpp" );
-		Code        string     = scan_file( "dependencies/gen.string.cpp" );
+		Code        string     = scan_file( "dependencies/string.cpp" );
-		Code        timing     = scan_file( "dependencies/gen.timing.cpp" );
+		Code        timing     = scan_file( "dependencies/timing.cpp" );
 
 		Builder
 		deps_impl;
@@ -115,18 +117,19 @@ int gen_main()
 
 	// gen.hpp
 	{
-		Code header_start = scan_file( "components/gen.header_start.hpp" );
+		Code header_start = scan_file( "components/header_start.hpp" );
 		Code nspace_macro = untyped_str( namespace_by_default ? nspace_default : nspace_non_default );
-		Code types        = scan_file( "components/gen.types.hpp" );
+		Code types        = scan_file( "components/types.hpp" );
-		Code data_structs = scan_file( "components/gen.data_structures.hpp" );
+		Code data_structs = scan_file( "components/data_structures.hpp" );
-		Code interface    = scan_file( "components/gen.interface.hpp" );
+		Code interface    = scan_file( "components/interface.hpp" );
-		Code header_end   = scan_file( "components/gen.header_end.hpp" );
+		Code header_end   = scan_file( "components/header_end.hpp" );
 
-		CodeBody ecode      = gen_ecode( "./components/ECode.csv" );
+		CodeBody ecode      = gen_ecode( "enums/ECode.csv" );
-		CodeBody eoperator  = gen_eoperator( "./components/EOperator.csv" );
+		CodeBody eoperator  = gen_eoperator( "enums/EOperator.csv" );
-		CodeBody especifier = gen_especifier( "./components/ESpecifier.csv" );
+		CodeBody especifier = gen_especifier( "enums/ESpecifier.csv" );
 
-		Code builder = scan_file( "filesystem/gen.builder.hpp" );
+		// TODO : Make this optional to include
+		Code builder = scan_file( "file_proecessors/builder.hpp" );
 
 		Builder
 		header;
@@ -159,18 +162,19 @@ int gen_main()
 	{
 		Code        impl_start      = scan_file( "components/gen.impl_start.cpp" );
 		CodeInclude header          = def_include( txt_StrC("gen.hpp") );
-		Code        data 	        = scan_file( "components/gen.data.cpp" );
+		Code        data 	        = scan_file( "components/static_data.cpp" );
-		Code        ast_case_macros = scan_file( "components/gen.ast_case_macros.cpp" );
+		Code        ast_case_macros = scan_file( "components/ast_case_macros.cpp" );
-		Code        ast			    = scan_file( "components/gen.ast.cpp" );
+		Code        ast			    = scan_file( "components/ast.cpp" );
-		Code        interface	    = scan_file( "components/gen.interface.cpp" );
+		Code        interface	    = scan_file( "components/interface.cpp" );
-		Code        upfront 	    = scan_file( "components/gen.interface.upfront.cpp" );
+		Code        upfront 	    = scan_file( "components/interface.upfront.cpp" );
-		Code 	    parsing 	    = scan_file( "components/gen.interface.parsing.cpp" );
+		Code 	    parsing 	    = scan_file( "components/interface.parsing.cpp" );
-		Code        untyped 	    = scan_file( "components/gen.untyped.cpp" );
+		Code        untyped 	    = scan_file( "components/untyped.cpp" );
 
-		CodeBody etoktype = gen_etoktype( "components/ETokType.csv", "components/AttributeTokens.csv" );
+		CodeBody etoktype = gen_etoktype( "enums/ETokType.csv", "enums/AttributeTokens.csv" );
 		CodeNamespace parser_nspace = def_namespace( name(Parser), def_namespace_body( args(etoktype)) );
 
-		Code builder = scan_file( "filesystem/gen.builder.cpp" );
+		// TODO : Make this optional to include
+		Code builder = scan_file( "file_proecessors/builder.cpp" );
 
 		Builder
 		impl;

project/gen.cpp

@@ -1,4 +1,4 @@
-#include "helpers/gen.push_ignores.inline.hpp"
+#include "helpers/push_ignores.inline.hpp"
 
 // ReSharper disable CppClangTidyClangDiagnosticSwitchEnum
 
@@ -16,19 +16,19 @@
 
 GEN_NS_BEGIN
 
-#include "components/gen.data.cpp"
+#include "components/static_data.cpp"
 
-#include "components/gen.ast_case_macros.cpp"
+#include "components/ast_case_macros.cpp"
-#include "components/gen.ast.cpp"
+#include "components/ast.cpp"
 
-#include "components/gen.interface.cpp"
+#include "components/interface.cpp"
-#include "components/gen.interface.upfront.cpp"
+#include "components/interface.upfront.cpp"
-#include "components/gen.etoktype.cpp"
+#include "components/etoktype.cpp"
-#include "components/gen.interface.parsing.cpp"
+#include "components/interface.parsing.cpp"
-#include "components/gen.untyped.cpp"
+#include "components/untyped.cpp"
 
-#include "filesystem/gen.builder.cpp"
+#include "file_proecessors/builder.cpp"
 
 GEN_NS_END
 
-#include "helpers/gen.pop_ignores.inline.hpp"
+#include "helpers/pop_ignores.inline.hpp"

project/gen.dep.cpp

@@ -1,19 +1,19 @@
 // This file is intended to be included within gen.cpp (There is no pragma diagnostic ignores)
 #include "gen.dep.hpp"
 
-#include "dependencies/gen.impl_start.cpp"
+#include "dependencies/impl_start.cpp"
 
 GEN_NS_BEGIN
 
-#include "dependencies/gen.debug.cpp"
+#include "dependencies/debug.cpp"
-#include "dependencies/gen.string_ops.cpp"
+#include "dependencies/string_ops.cpp"
-#include "dependencies/gen.printing.cpp"
+#include "dependencies/printing.cpp"
-#include "dependencies/gen.memory.cpp"
+#include "dependencies/memory.cpp"
-#include "dependencies/gen.parsing.cpp"
+#include "dependencies/parsing.cpp"
-#include "dependencies/gen.hashing.cpp"
+#include "dependencies/hashing.cpp"
-#include "dependencies/gen.string.cpp"
+#include "dependencies/string.cpp"
-#include "dependencies/gen.timing.cpp"
+#include "dependencies/timing.cpp"
 
-#include "dependencies/gen.file_handling.cpp"
+#include "dependencies/file_handling.cpp"
 
 GEN_NS_END

project/gen.dep.hpp

@@ -1,7 +1,7 @@
 // This file is intended to be included within gen.hpp (There is no pragma diagnostic ignores)
 #pragma once
 
-#include "dependencies/gen.header_start.hpp"
+#include "dependencies/header_start.hpp"
 
 #ifdef GEN_DONT_USE_NAMESPACE
 #	define GEN_NS_BEGIN
@@ -13,16 +13,17 @@
 
 GEN_NS_BEGIN
 
-#include "dependencies/gen.macros.hpp"
+#include "dependencies/macros.hpp"
-#include "dependencies/gen.basic_types.hpp"
+#include "dependencies/basic_types.hpp"
-#include "dependencies/gen.debug.hpp"
+#include "dependencies/debug.hpp"
-#include "dependencies/gen.memory.hpp"
+#include "dependencies/memory.hpp"
-#include "dependencies/gen.string_ops.hpp"
+#include "dependencies/string_ops.hpp"
-#include "dependencies/gen.printing.hpp"
+#include "dependencies/printing.hpp"
-#include "dependencies/gen.containers.hpp"
+#include "dependencies/containers.hpp"
-#include "dependencies/gen.string.hpp"
+#include "dependencies/string.hpp"
-#include "dependencies/gen.file_handling.hpp"
+#include "dependencies/parsing.hpp"
-#include "dependencies/gen.parsing.hpp"
+#include "dependencies/timing.hpp"
-#include "dependencies/gen.timing.hpp"
+
+#include "dependencies/file_handling.hpp"
 
 GEN_NS_END

project/gen.hpp

@@ -8,8 +8,8 @@
 */
 #pragma once
 
-#include "helpers/gen.push_ignores.inline.hpp"
+#include "helpers/push_ignores.inline.hpp"
-#include "components/gen.header_start.hpp"
+#include "components/header_start.hpp"
 
 #ifdef GEN_DONT_USE_NAMESPACE
 #	define GEN_NS_BEGIN
@@ -21,16 +21,16 @@
 
 GEN_NS_BEGIN
 
-#include "components/gen.types.hpp"
+#include "components/types.hpp"
-#include "components/gen.ecode.hpp"
+#include "components/ecode.hpp"
-#include "components/gen.eoperator.hpp"
+#include "components/eoperator.hpp"
-#include "components/gen.especifier.hpp"
+#include "components/especifier.hpp"
-#include "components/gen.data_structures.hpp"
+#include "components/data_structures.hpp"
-#include "components/gen.interface.hpp"
+#include "components/interface.hpp"
-#include "components/gen.header_end.hpp"
+#include "components/header_end.hpp"
 
-#include "filesystem/gen.builder.hpp"
+#include "file_processors/builder.hpp"
 
 GEN_NS_END
 
-#include "helpers/gen.pop_ignores.inline.hpp"
+#include "helpers/pop_ignores.inline.hpp"

scripts/Readme.md

@@ -1,12 +1,34 @@
 # Scripts
 
-Build and cleanup scripts for the test directory are found here along with `natvis` and `natstepfilter` files for debugging.
+Generation, testing, and cleanup scripts for the test directory are found here along with `natvis` and `natstepfilter` files for debugging.
 
-The build works as follows:
+## Refactoring
 
-* Compile and run the meta-program, it will dump files to the `test/gen` directory.
+`refactor.ps1` Provides a way to run the [refactor](github.com/Ed94/refactor) program. It uses the `gencpp.refactor` script to complete a mass refactor of all content within the files of the specified within the script.
-* Format the files using clang-format
-* Build a program that uses some the generated definitions. (Have not done yet)
 
-The `test/gen` directory has the meson.build config for the meta-program
+Currently `refactor` only supports naive sort of *find and replace* feature set and will not be able to rename identifiers excluisvely to a specific context (such as only renaming member names of a specific struct, etc).
-The `test` directory has the one for the dependent-program.
+
+## Build & Run Scripts
+
+**`clean.ps1`**  
+Remove any generated content from the repository.
+
+**`bootstrap.ps1`**  
+Generate a version of gencpp where components are inlined directly to `gen.<hpp/cpp>` and `gen. <hpp/cpp>`  
+Any heavily preprocessed code is not inlined and are instead generated using the code in the `helpers` directory.
+
+**`singlheader.build.ps1`**  
+Generate a single-header version of the library where all code that would normally good in the usual four files (see bootstrap) are inlined into a single `gen.hpp` file.  
+As with the bootstrap, any heavily preprocessed code is not inlined and instead generated with helper code.
+
+**`test.gen.build.ps1`**  
+Build the metaprogram for generating the test code.
+
+**`test.gen.ps1`**  
+Build (if not already) the metaprogram for generating test code, then run it to generate code.
+
+**`test.build.ps1`**  
+Build and run metaprogram, build test program.
+
+**`test.run.ps1`**  
+Build and run metaprogram, build and run test program.

scripts/get_sources.ps1

@@ -1,12 +0,0 @@
-[string[]] $include = 'gen.cpp' #'*.c', '*.cc', '*.cpp'
-# [string[]] $exclude =
-
-$path_root = git rev-parse --show-toplevel
-$path_proj = Join-Path $path_root project
-
-$files = Get-ChildItem -Recurse -Path $path_proj -Include $include -Exclude $exclude
-
-$sources = $files | Select-Object -ExpandProperty FullName | Resolve-Path -Relative
-$sources = $sources.Replace( '\', '/' )
-
-return $sources