Updates to documentation

Readme.md

@@ -5,14 +5,18 @@ An attempt at simple staged metaprogramming for c/c++.
 The library API is a composition of code element constructors.  
 These build up a code AST to then serialize with a file builder.
 
-This code base attempts follow the [handmade philosophy](https://handmade.network/manifesto),  
+This code base attempts follow the [handmade philosophy](https://handmade.network/manifesto).
-its not meant to be a black box metaprogramming utility, it should be easy to intergrate into a user's project domain.
+
+Its not meant to be a black box metaprogramming utility, it should be easy to intergrate into a user's project domain.  
 
 ## Notes
 
-The project has reached an *alpha* state, all the current functionality works for the test cases but it will most likely break in many other cases.  
+This project is still in development (very much an alpha state), so expect bugs and missing features.  
+See [issues](https://github.com/Ed94/gencpp/issues) for a list of known bugs or todos.
 
-A `natvis` and `natstepfilter` are provided in the scripts directory.
+The library can already be used to generate code just fine, but the parser is where the most work is needed. If your C++ isn't "down to earth" expect issues.
+
+A `natvis` and `natstepfilter` are provided in the scripts directory (its outdated, I'll update this readme when its not).
 
 ***The editor and scanner have not been implemented yet. The scanner will come first, then the editor.***
 
@@ -123,7 +127,7 @@ struct ArrayHeader
 ```
 
 **Note: The formatting shown here is not how it will look. For your desired formatting its recommended to run a pass through the files with an auto-formatter.**
-*(The library currently uses clang-format for formatting, beaware its pretty slow...)*
+*(The library currently uses clang-format for formatting, beware its pretty slow...)*
 
 ## Building
 

docs/Parsing.md

@@ -60,6 +60,7 @@ Exceptions:
 * typedefs allow for a preprocessed macro: `typedef MACRO();`
   * Disable with: `#define GEN_PARSER_DISABLE_MACRO_TYPEDEF`
 
+*(Exceptions are added on an on-demand basis)*
 *(See functions `parse_operator_function_or_variable` and `parse_typedef` )*
 
 Adding your own exceptions is possible by simply modifying the parser to allow for the syntax you need.

docs/Readme.md

@@ -32,7 +32,7 @@ Two generic templated containers are used throughout the library:
 * `template< class Type> struct Array`
 * `template< class Type> struct HashTable`
 
-Both Code and AST definitions have a `template< class Type> Code/AST cast()`. Its just an alternative way to explicitly cast to each other.
+Both Code and AST definitions have a `template< class Type> Code/AST :: cast()`. Its just an alternative way to explicitly cast to each other.
 
 `template< class Type> swap( Type& a, Type& b)` is used over a macro.
 
@@ -70,28 +70,36 @@ Data layout of AST struct:
 union {
     struct
     {
+        AST*      InlineCmt;       // Class, Constructor, Destructor, Enum, Friend, Functon, Operator, OpCast, Struct, Typedef, Using, Variable
         AST*      Attributes;      // Class, Enum, Function, Struct, Typedef, Union, Using, Variable
-        AST*      Specs;           // Function, Operator, Type symbol, Variable
+        AST*      Specs;           // Destructor, Function, Operator, Typename, Variable
         union {
-            AST*  InitializerList; // Constructor, Destructor
+            AST*  InitializerList; // Constructor
-            AST*  ParentType;      // Class, Struct
+            AST*  ParentType;      // Class, Struct, ParentType->Next has a possible list of interfaces.
-            AST*  ReturnType;      // Function, Operator
+            AST*  ReturnType;      // Function, Operator, Typename
             AST*  UnderlyingType;  // Enum, Typedef
             AST*  ValueType;       // Parameter, Variable
         };
         union {
-            AST*  BitfieldSize;    // Varaiable (Class/Struct Data Member)
+            AST*  BitfieldSize;    // Variable (Class/Struct Data Member)
-            AST*  Params;          // Function, Operator, Template
+            AST*  Params;          // Constructor, Function, Operator, Template, Typename
         };
         union {
-            AST*  ArrExpr;         // Type Symbol
+            AST*  ArrExpr;          // Typename
-            AST*  Body;            // Class, Constructr, Destructor, Enum, Function, Namespace, Struct, Union
+            AST*  Body;             // Class, Constructr, Destructor, Enum, Function, Namespace, Struct, Union
-            AST*  Declaration;     // Friend, Template
+            AST*  Declaration;      // Friend, Template
-            AST*  Value;           // Parameter, Variable
+            AST*  Value;            // Parameter, Variable
+        };
+        union {
+            AST*  NextVar;          // Variable; Possible way to handle comma separated variables declarations. ( , NextVar->Specs NextVar->Name NextVar->ArrExpr = NextVar->Value )
+            AST*  SpecsFuncSuffix;  // Only used with typenames, to store the function suffix if typename is function signature.
         };
     };
-    StringCached  Content;                     // Attributes, Comment, Execution, Include
+    StringCached  Content;          // Attributes, Comment, Execution, Include
-    SpecifierT    ArrSpecs[AST::ArrSpecs_Cap]; // Specifiers
+    struct {
+        SpecifierT ArrSpecs[AST::ArrSpecs_Cap]; // Specifiers
+        AST*       NextSpecs;                   // Specifiers
+    };
 };
 union {
     AST* Prev;
@@ -107,11 +115,13 @@ StringCached      Name;
 CodeT             Type;
 ModuleFlag        ModuleFlags;
 union {
-    b32           IsFunction; // Used by typedef to not serialize the name field.
+    b32           IsFunction;  // Used by typedef to not serialize the name field.
+    b32           IsParamPack; // Used by typename to know if type should be considered a parameter pack.
     OperatorT     Op;
     AccessSpec    ParentAccess;
     s32           NumEntries;
 };
+s32               Token;       // Handle to the token, stored in the CodeFile (Otherwise unretrivable)
 ```
 
 *`CodeT` is a typedef for `ECode::Type` which has an underlying type of `u32`*  
@@ -142,6 +152,13 @@ Data Notes:
 
 * The allocator definitions used are exposed to the user incase they want to dictate memory usage
   * You'll find the memory handling in `init`, `deinit`, `reset`, `gen_string_allocator`, `get_cached_string`, `make_code`.
+  * Allocators are defined with the `AllocatorInfo` structure found in `dependencies\memory.hpp`
+  * Most of the work is just defining the allocation procedure:
+
+```cpp
+    void* ( void* allocator_data, AllocType type, sw size, sw alignment, void* old_memory, sw old_size, u64 flags );
+```
+
 * ASTs are wrapped for the user in a Code struct which is a wrapper for a AST* type.
 * Both AST and Code have member symbols but their data layout is enforced to be POD types.
 * This library treats memory failures as fatal.
@@ -233,6 +250,7 @@ Interface :``
 * def_operator_cast
 * def_param
 * def_params
+* def_pragma
 * def_preprocess_cond
 * def_specifier
 * def_specifiers
@@ -382,8 +400,8 @@ The following are provided predefined by the library as they are commonly used:
 * `module_global_fragment`
 * `module_private_fragment`
 * `fmt_newline`
-* `pragma_once`
 * `param_varaidc` (Used for varadic definitions)
+* `pragma_once`
 * `preprocess_else`
 * `preprocess_endif`
 * `spec_const`
@@ -392,6 +410,7 @@ The following are provided predefined by the library as they are commonly used:
 * `spec_constinit`
 * `spec_extern_linkage` (extern)
 * `spec_final`
+* `spec_forceinline`
 * `spec_global` (global macro)
 * `spec_inline`
 * `spec_internal_linkage` (internal macro)
@@ -429,8 +448,8 @@ Optionally the following may be defined if `GEN_DEFINE_LIBRARY_CODE_CONSTANTS` i
 * `t_u16`
 * `t_u32`
 * `t_u64`
-* `t_sw`
+* `t_sw` (ssize_t)
-* `t_uw`
+* `t_uw` (size_t)
 * `t_f32`
 * `t_f64`
 
@@ -458,50 +477,10 @@ Editor and Scanner are disabled by default, use `GEN_FEATURE_EDITOR` and `GEN_FE
 ### Builder is a similar object to the jai language's string_builder
 
 * The purpose of it is to generate a file.
-* A file is specified and opened for writing using the open( file_path) ) function.
+* A file is specified and opened for writing using the open( file_path) function.
 * The code is provided via print( code ) function  will be serialized to its buffer.
 * When all serialization is finished, use the write() command to write the buffer to the file.
 
-### Editor is for editing a series of files/asts based on a set of requests provided to it
+### Scanner Auxillary Interface
 
-**Note: Not implemented yet**
+Provides *(eventually)* `scan_file` to automatically populate a CodeFile which contains a parsed AST (`Code`) of the file, with any contextual failures that are reported from the parser.
-
-* The purpose is to overrite a specific file, it places its contents in a buffer to scan.
-* If editing an AST it will generate a new ast as a result (ASTs are not edited).
-* Requests are populated using the following interface:
-  * add : Add code.
-  * remove : Remove code.
-  * replace: Replace code.
-
-All three have the same parameters with exception to remove which only has SymbolInfo and Policy:
-
-* SymbolInfo:
-  * File      : The file the symbol resides in. Leave null to indicate to search all files. Leave null to indicated all-file search.
-  * Marker    : #define symbol that indicates a location or following signature is valid to manipulate. Leave null to indicate the signature should only be used.
-  * Signature : Use a Code symbol to find a valid location to manipulate, can be further filtered with the marker. Leave null to indicate the marker should only be used.
-
-* Policy : Additional policy info for completing the request (empty for now)
-* Code   : Code to inject if adding, or replace existing code with.
-
-Additionally if `GEN_FEATURE_EDITOR_REFACTOR` is defined, refactor( file_path, specification_path ) wil be made available.  
-Refactor is based of the refactor library and uses its interface.  
-It will on call add a request to the queue to run the refactor script on the file.
-
-### Scanner allows the user to sift through a series of files/asts based on a set of requests provided to it
-
-**Note: Not implemented yet**
-
-* The purpose is to grab definitions to generate metadata or generate new code from these definitions.
-* Requests are populated using the add( SymbolInfo, Policy ) function. The symbol info is the same as the one used for the editor. So is the case with Policy.
-
-The file will only be read from, no writing supported.
-
-### Additional Info (Editor and Scanner)
-
-When all requests have been populated, call process_requests().  
-It will provide an output of receipt data of the results when it completes.
-
-Files may be added to the Editor and Scanner additionally with add_files( num, files ).  
-This is intended for when you have requests that are for multiple files.
-
-Request queue in both Editor and Scanner are cleared once process_requests completes.

project/Readme.md

@@ -8,8 +8,7 @@ They contain includes for its various components: `components/<component_name>.<
 Dependencies are bundled into `gen.dep.<hpp/cpp>`.  
 Just like the `gen.<hpp/cpp>` they include their components: `dependencies/<dependency_name>.<hpp/cpp>`
 
-The fle processors are in their own respective files. (Ex: `file_processors/<file_processor>.<hpp/cpp>` )  
+Code not making up the core library is located in `auxiliary/<auxiliary_name>.<hpp/cpp>`. These are optional extensions or tools for the library.
-They directly include `depedencies/file_handling.<hpp/cpp>` as the core library does not include file processing by defualt.
 
 **TODO : Right now the library is not finished, as such the first self-hosting iteration is still WIP**  
 Both libraries use *pre-generated* (self-hosting I guess) version of the library to then generate the latest version of itself.  
@@ -30,10 +29,11 @@ Feature Macros:
 * `GEN_ENFORCE_STRONG_CODE_TYPES` : Enforces casts to filtered code types.
 * `GEN_EXPOSE_BACKEND` : Will expose symbols meant for internal use only.
 * `GEN_ROLL_OWN_DEPENDENCIES` : Optional override so that user may define the dependencies themselves.
+* `GEN_DONT_ALLOW_INVALID_CODE` (Not implemented yet) : Will fail when an invalid code is constructed, parsed, or serialized.
 
 ## On multi-threading
 
-Currently unsupported.
+Currently unsupported. I want the library to be *stable* and *correct*, with the addition of exhausting all basic single-threaded optimizations before I consider multi-threading.
 
 ## Extending the library
 
@@ -51,5 +51,5 @@ Names or Content fields are interned strings and thus showed be cached using `ge
 `def_operator` is the most sophisticated constructor as it has multiple permutations of definitions that could be created that are not trivial to determine if valid.
 
 The library has its code segmented into component files, use it to help create a derived version without needing to have to rewrite a generated file directly or build on top of the header via composition or inheritance.
-When the scanner is implemented, this will be even easier to customize.
 
+The parser is documented under `docs/Parsing.md` and `docs/Parser_Algo.md`.