Reorganization of parser, refactor of parse_type( bool* ) and progression of parser docs

Wanted to make parser implementation easier to sift through, so I emphasized alphabetical order more.

Since I couldn't just strip whitespace from typenames I decided to make the parse_type more aware of the typename's components if it was a function signature.
This ofc lead to the dark & damp hell that is parsing typenames.

Also made initial implementation to support parsing decltype within a typename signature..

The test failure for the singleheader is still a thing, these changes have not addressed that.

.vscode/settings.json

@@ -24,7 +24,9 @@
 		"filesystem": "cpp",
 		"format": "cpp",
 		"ratio": "cpp",
-		"xstring": "cpp"
+		"xstring": "cpp",
+		"functional": "cpp",
+		"vector": "cpp"
 	},
 	"C_Cpp.intelliSenseEngineFallback": "disabled",
 	"mesonbuild.configureOnOpen": true,

docs/ASTs.md

@@ -561,6 +561,8 @@ Fields:
 ```cpp
 CodeAttributes Attributes;
 CodeSpecifiers Specs;
+CodeReturnType ReturnType;
+CodeParam      Params;
 Code           ArrExpr;
 Code           Prev;
 Code           Next;

docs/Parser_Algo.md

@@ -0,0 +1,91 @@
+# Parser's Algorithim
+
+gencpp uses a hand-written recursive descent parser. Both the lexer and parser handle a full C/C++ file in a single pass.
+
+## Notable implementation background
+
+### Lexer
+
+The lex procedure does the lexical pass of content provided as a `StrC` type.
+The tokens are stored (for now) in `gen::Parser::Tokens`.
+
+Fields:
+```cpp
+Array<Token> Arr;
+s32          Idx;
+```
+
+
+What token types are supported can be found in [ETokType.csv](../project/enums/ETokType.csv) you can also find the token types in [ETokType.h](../project/components/gen/etoktype.cpp) , which is the generated enum from the csv file.
+
+Tokens are defined with the struct `gen::Parser::Token`:
+
+Fields:
+```cpp
+char const* Text;
+sptr        Length;
+TokType     Type;
+s32         Line;
+s32         Column;
+bool 	    IsAssign;
+```
+
+`IsAssign` is a flag that is set when the token is an assignment operator. Which is used for various purposes:
+
+* Using statment assignment
+* Parameter argument default value assignment
+* Variable declaration initialization assignment
+
+I plan to replace IsAssign with a general flags field and properly keep track of all operator types instead of abstracting it away to `ETokType::Operator`.
+
+Traversing the tokens is done with the following interface macros:
+
+| Macro | Description |
+| --- | --- |
+| `currtok_noskip` | Get the current token without skipping whitespace |
+| `currtok` | Get the current token, skip any whitespace tokens |
+| `prevtok` | Get the previous token (does not skip whitespace) |
+| `nexttok` | Get the next token (does not skip whitespace) |
+| `eat( Token Type )` | Check to see if the current token is of the given type, if so, advance Token's index to the next token |
+| `left` | Get the number of tokens left in the token array |
+| `check_noskip` | Check to see if the current token is of the given type, without skipping whitespace |
+| `check` | Check to see if the current token is of the given type, skip any whitespace tokens |
+
+### Parser
+
+The parser has a limited user interface, only specific types of definitions or statements are expected to be provided by the user directly when using to construct an AST dynamically (See SOA for example). It however does attempt to provide capability to parse a full C/C++ from production codebases. 
+
+Each public user interface procedure has the following format:
+
+```cpp
+CodeStruct parse_<definition type>( StrC def )
+{
+    check_parse_args( def );
+    using namespace Parser;
+
+    TokArray toks = lex( def );
+    if ( toks.Arr == nullptr )
+        return CodeInvalid;
+
+    // Parse the tokens and return a constructed AST using internal procedures
+    ...
+}
+```
+
+The most top-level parsing procedure used for C/C++ file parsing is `parse_global_body`:
+
+It uses a helper procedure called `parse_global_nspace`.
+
+Each internal procedure will be 
+
+## parse_global_nspace
+
+1. Make sure the type provided to the helper function is a `Namespace_Body`, `Global_Body`, `Export_Body`, `Extern_Linkage_body`.
+2. If its not a `Global_Body` eat the opening brace for the scope.
+3. 
+
+
+## parse_type
+
+This is the worst part of the parser. Because other than actual expression values in C++, typenames are the second worst thing to parse in the langauge.
+

docs/Parsing.md

@@ -1,11 +1,12 @@
 # Parsing
 
 The library features a naive parser tailored for only what the library needs to construct the supported syntax of C++ into its AST.
+
 This parser does not, and should not do the compiler's job. By only supporting this minimal set of features, the parser is kept (so far) around 5000 loc.
 
 You can think of this parser of a frontend parser vs a semantic parser. Its intuitively similar to WYSIWYG. What you precerive as the syntax from the user-side before the compiler gets a hold of it, is what you get.
 
-The parsing implementation supports the following for the user:
+User exposed interface:
 
 ```cpp
 CodeClass       parse_class        ( StrC class_def       );
@@ -55,9 +56,9 @@ Any preprocessor definition abuse that changes the syntax of the core language i
 Exceptions:
 
 * function signatures are allowed for a preprocessed macro: `neverinline MACRO() { ... }`
-    * Disable with: `#define GEN_PARSER_DISABLE_MACRO_FUNCTION_SIGNATURES`
+  * Disable with: `#define GEN_PARSER_DISABLE_MACRO_FUNCTION_SIGNATURES`
 * typedefs allow for a preprocessed macro: `typedef MACRO();`
-    * Disable with: `#define GEN_PARSER_DISABLE_MACRO_TYPEDEF`
+  * Disable with: `#define GEN_PARSER_DISABLE_MACRO_TYPEDEF`
 
 *(See functions `parse_operator_function_or_variable` and `parse_typedef` )*
 
@@ -75,8 +76,6 @@ The lexing and parsing takes shortcuts from whats expected in the standard.
   * Assumed to *come before specifiers* (`const`, `constexpr`, `extern`, `static`, etc) for a function
   * Or in the usual spot for class, structs, (*right after the declaration keyword*)
   * typedefs have attributes with the type (`parse_type`)
-* As a general rule; if its not available from the upfront constructors, its not available in the parsing constructors.
-  * *Upfront constructors are not necessarily used in the parsing constructors, this is just a good metric to know what can be parsed.*
 * Parsing attributes can be extended to support user defined macros by defining `GEN_DEFINE_ATTRIBUTE_TOKENS` (see `gen.hpp` for the formatting)
 
 Empty lines used throughout the file are preserved for formatting purposes during ast serialization.

project/components/ast.cpp

@@ -773,7 +773,8 @@ String AST::to_string()
 
 			result.append( "typedef ");
 
-			if ( IsFunction )
+			// Determines if the typedef is a function typename
+			if ( UnderlyingType->ReturnType )
 				result.append( UnderlyingType->to_string() );
 			else
 				result.append_fmt( "%S %S", UnderlyingType->to_string(), Name );
@@ -796,21 +797,45 @@ String AST::to_string()
 
 		case Typename:
 		{
-			if ( Attributes || Specs )
+		#if GEN_USE_NEW_TYPENAME_PARSING
+			if ( ReturnType && Params )
 			{
 				if ( Attributes )
 					result.append_fmt( "%S ", Attributes->to_string() );
-
-				if ( Specs )
-					result.append_fmt( "%S %S", Name, Specs->to_string() );
-
 				else
-					result.append_fmt( "%S", Name );
+				{
+					if ( Specs )
+						result.append_fmt( "%S ( %S ) ( %S ) %S", ReturnType->to_string(), Name, Params->to_string(), Specs->to_string() );
+					else
+						result.append_fmt( "%S ( %S ) ( %S )", ReturnType->to_string(), Name, Params->to_string() );
+				}
+
+				break;
 			}
-			else
+		#else
+			if ( ReturnType && Params )
 			{
-				result.append_fmt( "%S", Name );
+				if ( Attributes )
+					result.append_fmt( "%S ", Attributes->to_string() );
+				else
+				{
+					if ( Specs )
+						result.append_fmt( "%S %S ( %S ) %S", ReturnType->to_string(), Name, Params->to_string(), Specs->to_string() );
+					else
+						result.append_fmt( "%S %S ( %S )", ReturnType->to_string(), Name, Params->to_string() );
+				}
+
+				break;
 			}
+		#endif
+
+			if ( Attributes )
+				result.append_fmt( "%S ", Attributes->to_string() );
+
+			if ( Specs )
+				result.append_fmt( "%S %S", Name, Specs->to_string() );
+			else
+				result.append_fmt( "%S", Name );
 
 			if ( IsParamPack )
 				result.append("...");

project/components/ast.hpp

@@ -6,11 +6,6 @@
 #include "gen/especifier.hpp"
 #endif
 
-namespace Parser
-{
-	struct Token;
-}
-
 struct AST;
 struct AST_Body;
 struct AST_Attributes;
@@ -234,12 +229,15 @@ struct AST
 	union {
 		struct
 		{
-			AST*      InlineCmt;      // Class, Constructor, Destructor, Enum, Friend, Functon, Operator, OpCast, Struct, Typedef, Using, Variable
-			AST*      Attributes;     // Class, Enum, Function, Struct, Typedef, Union, Using, Variable
-			AST*      Specs;          // Destructor, Function, Operator, Typename, Variable
+			union {
+				AST*  InlineCmt;       // Class, Constructor, Destructor, Enum, Friend, Functon, Operator, OpCast, Struct, Typedef, Using, Variable
+				AST*  SpecsFuncSuffix; // Only used with typenames, to store the function suffix if typename is function signature.
+			};
+			AST*      Attributes;      // Class, Enum, Function, Struct, Typedef, Union, Using, Variable
+			AST*      Specs;           // Destructor, Function, Operator, Typename, Variable
 			union {
 				AST*  InitializerList; // Constructor
-				AST*  ParentType;      // Class, Struct
+				AST*  ParentType;      // Class, Struct, ParentType->Next has a possible list of interfaces.
 				AST*  ReturnType;      // Function, Operator
 				AST*  UnderlyingType;  // Enum, Typedef
 				AST*  ValueType;       // Parameter, Variable
@@ -249,13 +247,13 @@ struct AST
 				AST*  Params;          // Constructor, Function, Operator, Template
 			};
 			union {
-				AST*  ArrExpr;        // Typename
-				AST*  Body;           // Class, Constructr, Destructor, Enum, Function, Namespace, Struct, Union
-				AST*  Declaration;    // Friend, Template
-				AST*  Value;          // Parameter, Variable
+				AST*  ArrExpr;          // Typename
+				AST*  Body;             // Class, Constructr, Destructor, Enum, Function, Namespace, Struct, Union
+				AST*  Declaration;      // Friend, Template
+				AST*  Value;            // Parameter, Variable
 			};
 		};
-		StringCached  Content;        // Attributes, Comment, Execution, Include
+		StringCached  Content;          // Attributes, Comment, Execution, Include
 		SpecifierT    ArrSpecs[AST::ArrSpecs_Cap]; // Specifiers
 	};
 	union {
@@ -286,12 +284,15 @@ struct AST_POD
 	union {
 		struct
 		{
-			AST*      InlineCmt;      // Class, Constructor, Destructor, Enum, Friend, Functon, Operator, OpCast, Struct, Typedef, Using, Variable
-			AST*      Attributes;     // Class, Enum, Function, Struct, Typename, Union, Using, Variable
-			AST*      Specs;          // Function, Operator, Typename, Variable
+			union {
+				AST*  InlineCmt;       // Class, Constructor, Destructor, Enum, Friend, Functon, Operator, OpCast, Struct, Typedef, Using, Variable
+				AST*  SpecsFuncSuffix; // Only used with typenames, to store the function suffix if typename is function signature.
+			};
+			AST*      Attributes;      // Class, Enum, Function, Struct, Typename, Union, Using, Variable
+			AST*      Specs;           // Function, Operator, Typename, Variable
 			union {
 				AST*  InitializerList; // Constructor
-				AST*  ParentType;      // Class, Struct
+				AST*  ParentType;      // Class, Struct, ParentType->Next has a possible list of interfaces.
 				AST*  ReturnType;      // Function, Operator
 				AST*  UnderlyingType;  // Enum, Typedef
 				AST*  ValueType;       // Parameter, Variable
@@ -301,13 +302,13 @@ struct AST_POD
 				AST*  Params;          // Function, Operator, Template
 			};
 			union {
-				AST*  ArrExpr;        // Type Symbol
-				AST*  Body;           // Class, Constructr, Destructor, Enum, Function, Namespace, Struct, Union
-				AST*  Declaration;    // Friend, Template
-				AST*  Value;          // Parameter, Variable
+				AST*  ArrExpr;         // Type Symbol
+				AST*  Body;            // Class, Constructr, Destructor, Enum, Function, Namespace, Struct, Union
+				AST*  Declaration;     // Friend, Template
+				AST*  Value;           // Parameter, Variable
 			};
 		};
-		StringCached  Content;        // Attributes, Comment, Execution, Include
+		StringCached  Content;         // Attributes, Comment, Execution, Include
 		SpecifierT    ArrSpecs[AST::ArrSpecs_Cap]; // Specifiers
 	};
 	union {

project/components/ast_types.hpp

@@ -472,11 +472,11 @@ struct AST_Type
 		char 		       _PAD_[ sizeof(SpecifierT) * AST::ArrSpecs_Cap ];
 		struct
 		{
-			char           _PAD_CMT_[ sizeof(AST*) ];
+			CodeSpecifiers SpecsFuncSuffix; // Only used for function signatures
 			CodeAttributes Attributes;
 			CodeSpecifiers Specs;
-			CodeType       ReturnType; // Only used for function signatures
-			CodeParam      Params;     // Only used for function signatures
+			CodeType       ReturnType;      // Only used for function signatures
+			CodeParam      Params;          // Only used for function signatures
 			Code           ArrExpr;
 		};
 	};

project/components/gen/especifier.hpp

@@ -31,6 +31,7 @@ namespace ESpecifier
 		Virtual,
 		Const,
 		Final,
+		NoExceptions,
 		Override,
 		Pure,
 		NumSpecifiers
@@ -67,6 +68,7 @@ namespace ESpecifier
 			{ sizeof( "virtual" ),       "virtual"      },
 			{ sizeof( "const" ),         "const"        },
 			{ sizeof( "final" ),         "final"        },
+			{ sizeof( "noexcept" ),      "noexcept"     },
 			{ sizeof( "override" ),      "override"     },
 			{ sizeof( "= 0" ),           "= 0"          },
 		};

project/enums/ESpecifier.csv

@@ -21,5 +21,6 @@ Volatile,         volatile
 Virtual,          virtual
 Const,            const
 Final,            final
+NoExceptions,     noexcept
 Override,         override
 Pure,             = 0