Merge pull request #9 from Ed94/Preprocessor_support

Preprocessor support

Readme.md

@@ -8,12 +8,6 @@ These build up a code AST to then serialize with a file builder.
 This code base attempts follow the [handmade philosophy](https://handmade.network/manifesto),  
 its not meant to be a black box metaprogramming utility, its meant for the user to extend for their project domain.
 
-### TOC
-
-* [Notes](#notes)
-* [Usage](#usage)
-* [Building](#building)
-
 ## Notes
 
 The project has reached an *alpha* state, all the current functionality works for the test cases but it will most likely break in many other cases.  
@@ -23,7 +17,7 @@ A `natvis` and `natstepfilter` are provided in the scripts directory.
 
 ***The editor and scanner have not been implemented yet. The scanner will come first, then the editor.***
 
-A C variant is hosted [here](https://github.com/Ed94/genc); I haven't gotten headwind on it, should be easier to make than this...
+A C variant is hosted [here](https://github.com/Ed94/genc); I will complete it when this library is feature complete, it should be easier to make than this...
 
 ## Usage
 

docs/Parsing.md

@@ -3,26 +3,6 @@
 The library features a naive parser tailored for only what the library needs to construct the supported syntax of C++ into its AST.  
 This parser does not, and should not do the compiler's job. By only supporting this minimal set of features, the parser is kept under 5000 loc.
 
-
-Everything is done in one pass for both the preprocessor directives and the rest of the language.  
-The parser performs no macro expansion as the scope of gencpp feature-set is to only support the preprocessor for the goal of having rudimentary awareness of preprocessor ***conditionals***,  ***defines***, and ***includes***.  
-*(Conditionals and defines are a TODO)*
-
-The keywords supported for the preprocessor are:
-
-* include
-* define
-* if
-* ifdef
-* elif
-* endif
-* undef
-
-Just like with actual preprocessor, each directive # line is considered one preproecessor unit, and will be treated as one Preprocessor AST. *These ASTs will be considered members or entries of braced scope they reside within*.
-All keywords except *include* are suppported as members of a scope for a class/struct, global, or namespace body.  
-
-Any preprocessor definition abuse that changes the syntax of the core language is unsupported and will fail to parse if not kept within an execution scope (function body, or expression assignment).
-
 The parsing implementation supports the following for the user:
 
 ```cpp
@@ -47,3 +27,26 @@ CodeVar       parse_variable     ( StrC var_def       );
 
 ***Parsing will aggregate any tokens within a function body or expression statement to an untyped Code AST.***
 
+Everything is done in one pass for both the preprocessor directives and the rest of the language.  
+The parser performs no macro expansion as the scope of gencpp feature-set is to only support the preprocessor for the goal of having rudimentary awareness of preprocessor ***conditionals***,  ***defines***, and ***includes***, and ***`pragmas`**.  
+
+The keywords supported for the preprocessor are:
+
+* include
+* define
+* if
+* ifdef
+* elif
+* endif
+* undef
+* pragma
+
+Each directive `#` line is considered one preproecessor unit, and will be treated as one Preprocessor AST. *These ASTs will be considered members or entries of braced scope they reside within*.
+All keywords except *include* are suppported as members of a scope for a class/struct, global, or namespace body.  
+
+Any preprocessor definition abuse that changes the syntax of the core language is unsupported and will fail to parse if not kept within an execution scope (function body, or expression assignment).
+
+Exceptions to the above rule (If its too hard to keep track of just follow the above notion):
+
+* Typedefs allow of a macro exansion to be defined after the keyword; Ex: `typedef GEN_FILE_OPEN_PROC( file_open_proc );`
+

docs/Readme.md

@@ -42,6 +42,7 @@ Otherwise the library is free of any templates.
 
 * Execution statement validation : Execution expressions are defined using the untyped AST.
   * Lambdas (This naturally means its unsupported)
+* Non-trivial template validation support.
 * RAII : This needs support for constructors/destructor parsing
   * Haven't gotten around to yet (its in the github issues)
 

gencpp.10x

@@ -8,8 +8,8 @@
 		<ShowEmptyFolders>true</ShowEmptyFolders>
 		<IsVirtual>false</IsVirtual>
 		<IsFolder>false</IsFolder>
-		<BuildCommand>powershell ./scripts/build.ps1</BuildCommand>
+		<BuildCommand>powershell ./scripts/test.gen_run.ps1</BuildCommand>
-		<RebuildCommand>powershell ./scripts/rebuild.ps1</RebuildCommand>
+		<RebuildCommand></RebuildCommand>
 		<BuildFileCommand></BuildFileCommand>
 		<CleanCommand>powershell ./scripts/clean.ps1</CleanCommand>
 		<BuildWorkingDirectory></BuildWorkingDirectory>
@@ -38,7 +38,9 @@
 			<AdditionalIncludePath>C:\Program Files (x86)\Windows Kits\10\\include\10.0.19041.0\\cppwinrt</AdditionalIncludePath>
 			<AdditionalIncludePath>C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um</AdditionalIncludePath>
 		</AdditionalIncludePaths>
-		<Defines></Defines>
+		<Defines>
+			<Define>GEN_TIME</Define>
+		</Defines>
 		<ConfigProperties>
 			<ConfigAndPlatform>
 				<Name>Debug:x64</Name>

project/Readme.md

@@ -12,7 +12,7 @@ Just like the `gen.<hpp/cpp>` they include their components: `dependencies/<depe
 The fle processors are in their own respective files. (Ex: `file_processors/<file_processor>.<hpp/cpp>` )  
 They directly include `depedencies/file_handling.<hpp/cpp>` as the core library does not include file processing by defualt.
 
-**TODO : Right now the library is not finished structurally, as such the first self-hosting iteration is still WIP**  
+**TODO : Right now the library is not finished, as such the first self-hosting iteration is still WIP**  
 Both libraries use *pre-generated* (self-hosting I guess) version of the library to then generate the latest version of itself.  
 (sort of a verification that the generated version is equivalent).
 

project/components/ast.cpp

@@ -38,10 +38,13 @@ String AST::to_string()
 
 		case Comment:
 		{
+			result.append("\n");
+
 			static char line[MaxCommentLineLength];
 
 			s32 left  = Content.length();
 			s32 index = 0;
+			s32 curr  = 0;
 			do
 			{
 				s32 length = 0;
@@ -49,12 +52,17 @@ String AST::to_string()
 				{
 					length++;
 					left--;
+					index++;
 				}
+				index++;
 
-				str_copy( line, Content, length );
+				str_copy( line, Content + curr, length );
-				line[length] = '\0';
+				result.append_fmt( "//%.*s", length, line );
+				mem_set( line, 0, MaxCommentLineLength);
 
-				result.append_fmt( "// %s", line );
+				length++;
+				left--;
+				curr = index;
 			}
 			while ( left--, left > 0 );
 		}
@@ -107,13 +115,16 @@ String AST::to_string()
 				}
 				else
 				{
-					result.append_fmt( "%s \n{\n%s\n};", Name, Body->to_string() );
+					result.append_fmt( "%s \n{\n%s\n}", Name, Body->to_string() );
 				}
 			}
 			else
 			{
-				result.append_fmt( "class %s\n{\n%s\n};", Name, Body->to_string() );
+				result.append_fmt( "class %s\n{\n%s\n}", Name, Body->to_string() );
 			}
+
+			if ( Parent == nullptr || ( Parent->Type != ECode::Typedef && Parent->Type != ECode::Variable ) )
+				result.append(";");
 		}
 		break;
 
@@ -123,9 +134,12 @@ String AST::to_string()
 				result.append( "export " );
 
 			if ( Attributes )
-				result.append_fmt( "class %s %s;", Attributes->to_string(), Name );
+				result.append_fmt( "class %s %s", Attributes->to_string(), Name );
 
-			else result.append_fmt( "class %s;", Name );
+			else result.append_fmt( "class %s", Name );
+
+			if ( Parent == nullptr || ( Parent->Type != ECode::Typedef && Parent->Type != ECode::Variable ) )
+				result.append(";");
 		}
 		break;
 
@@ -142,21 +156,24 @@ String AST::to_string()
 					result.append_fmt( "%s ", Attributes->to_string() );
 
 				if ( UnderlyingType )
-					result.append_fmt( "%s : %s\n{\n%s\n};"
+					result.append_fmt( "%s : %s\n{\n%s\n}"
 						, Name
 						, UnderlyingType->to_string()
 						, Body->to_string()
 					);
 
-				else result.append_fmt( "%s\n{\n%s\n};"
+				else result.append_fmt( "%s\n{\n%s\n}"
 					, Name
 					, Body->to_string()
 				);
 			}
-			else result.append_fmt( "enum %s\n{\n%s\n};"
+			else result.append_fmt( "enum %s\n{\n%s\n}"
 				, Name
 				, Body->to_string()
 			);
+
+			if ( Parent == nullptr || ( Parent->Type != ECode::Typedef && Parent->Type != ECode::Variable ) )
+				result.append(";");
 		}
 		break;
 
@@ -168,7 +185,10 @@ String AST::to_string()
 			if ( Attributes )
 				result.append_fmt( "%s ", Attributes->to_string() );
 
-			result.append_fmt( "enum %s : %s;", Name, UnderlyingType->to_string() );
+			result.append_fmt( "enum %s : %s", Name, UnderlyingType->to_string() );
+
+			if ( Parent == nullptr || ( Parent->Type != ECode::Typedef && Parent->Type != ECode::Variable ) )
+				result.append(";");
 		}
 		break;
 
@@ -188,7 +208,7 @@ String AST::to_string()
 
 				if ( UnderlyingType )
 				{
-					result.append_fmt( "%s : %s\n{\n%s\n};"
+					result.append_fmt( "%s : %s\n{\n%s\n}"
 						, Name
 						, UnderlyingType->to_string()
 						, Body->to_string()
@@ -196,7 +216,7 @@ String AST::to_string()
 				}
 				else
 				{
-					result.append_fmt( "%s\n{\n%s\n};"
+					result.append_fmt( "%s\n{\n%s\n}"
 						, Name
 						, Body->to_string()
 					);
@@ -204,10 +224,13 @@ String AST::to_string()
 			}
 			else
 			{
-				result.append_fmt( "enum class %s\n{\n%s\n};"
+				result.append_fmt( "enum class %s\n{\n%s\n}"
 					, Body->to_string()
 				);
 			}
+
+			if ( Parent == nullptr || ( Parent->Type != ECode::Typedef && Parent->Type != ECode::Variable ) )
+				result.append(";");
 		}
 		break;
 
@@ -221,7 +244,10 @@ String AST::to_string()
 			if ( Attributes )
 				result.append_fmt( "%s ", Attributes->to_string() );
 
-			result.append_fmt( "%s : %s;", Name, UnderlyingType->to_string() );
+			result.append_fmt( "%s : %s", Name, UnderlyingType->to_string() );
+
+			if ( Parent == nullptr || ( Parent->Type != ECode::Typedef && Parent->Type != ECode::Variable ) )
+				result.append(";");
 		}
 		break;
 
@@ -264,7 +290,7 @@ String AST::to_string()
 				result.append_fmt( "%s ", Attributes->to_string() );
 
 			if ( Specs )
-				result.append_fmt( "%s\n", Specs->to_string() );
+				result.append_fmt( "%s", Specs->to_string() );
 
 			if ( ReturnType )
 				result.append_fmt( "%s %s(", ReturnType->to_string(), Name );
@@ -304,7 +330,7 @@ String AST::to_string()
 				result.append_fmt( "%s ", Attributes->to_string() );
 
 			if ( Specs )
-				result.append_fmt( "%s\n", Specs->to_string() );
+				result.append_fmt( "%s", Specs->to_string() );
 
 			if ( ReturnType )
 				result.append_fmt( "%s %s(", ReturnType->to_string(), Name );
@@ -430,7 +456,10 @@ String AST::to_string()
 		{
 			if ( Specs )
 			{
-				result.append_fmt( "operator %s()" );
+				if ( Name && Name.length() )
+					result.append_fmt( "%.*soperator %s()", Name.length(), Name, EOperator::to_str( Op ));
+				else
+					result.append_fmt( "operator %s()", EOperator::to_str( Op ) );
 
 				CodeSpecifiers specs = cast<CodeSpecifiers>();
 
@@ -444,6 +473,9 @@ String AST::to_string()
 				break;
 			}
 
+			if ( Name && Name.length() )
+				result.append_fmt("%.*soperator %s()\n{\n%s\n}", Name.length(), Name, ValueType->to_string(), Body->to_string() );
+			else
 				result.append_fmt("operator %s()\n{\n%s\n}", ValueType->to_string(), Body->to_string() );
 		}
 		break;
@@ -451,7 +483,7 @@ String AST::to_string()
 		case Operator_Cast_Fwd:
 			if ( Specs )
 			{
-				result.append_fmt( "operator %s()" );
+				result.append_fmt( "operator %s()",  ValueType->to_string() );
 
 				CodeSpecifiers specs = cast<CodeSpecifiers>();
 
@@ -461,7 +493,7 @@ String AST::to_string()
 						result.append_fmt( " %s", (char const*)ESpecifier::to_str( spec ) );
 				}
 
-				result.append_fmt( ";", Body->to_string() );
+				result.append( ";" );
 				break;
 			}
 
@@ -470,6 +502,12 @@ String AST::to_string()
 
 		case Parameters:
 		{
+			if ( ValueType == nullptr )
+			{
+				result.append_fmt( "%s", Name );
+				break;
+			}
+
 			if ( Name )
 				result.append_fmt( "%s %s", ValueType->to_string(), Name );
 
@@ -489,8 +527,40 @@ String AST::to_string()
 		}
 		break;
 
-		case Preprocessor_Include:
+		case Preprocess_Define:
-			result.append_fmt( "#include \"%s\"", Name );
+			result.append_fmt( "#define %s%s", Name, Content );
+		break;
+
+		case Preprocess_If:
+			result.append_fmt( "#if %s", Content );
+		break;
+
+		case Preprocess_IfDef:
+			result.append_fmt( "#ifdef %s", Content );
+		break;
+
+		case Preprocess_IfNotDef:
+			result.append_fmt( "#ifndef %s", Content );
+		break;
+
+		case Preprocess_Include:
+			result.append_fmt( "#include \"%s\"", Content );
+		break;
+
+		case Preprocess_ElIf:
+			result.append_fmt( "#elif %s", Content );
+		break;
+
+		case Preprocess_Else:
+			result.append_fmt( "#else" );
+		break;
+
+		case Preprocess_EndIf:
+			result.append_fmt( "#endif\n" );
+		break;
+
+		case Preprocess_Pragma:
+			result.append_fmt( "#pragma %s", Content );
 		break;
 
 		case Specifiers:
@@ -555,13 +625,16 @@ String AST::to_string()
 				{
 					if ( Name )
 
-					result.append_fmt( "%s \n{\n%s\n};", Name, Body->to_string() );
+					result.append_fmt( "%s \n{\n%s\n}", Name, Body->to_string() );
 				}
 			}
 			else
 			{
-				result.append_fmt( "struct %s\n{\n%s\n};", Name, Body->to_string() );
+				result.append_fmt( "struct %s\n{\n%s\n}", Name, Body->to_string() );
 			}
+
+			if ( Parent == nullptr || ( Parent->Type != ECode::Typedef && Parent->Type != ECode::Variable ) )
+				result.append(";");
 		}
 		break;
 
@@ -571,9 +644,12 @@ String AST::to_string()
 				result.append( "export " );
 
 			if ( Attributes )
-				result.append_fmt( "struct %s %s;", Attributes->to_string(), Name );
+				result.append_fmt( "struct %s %s", Attributes->to_string(), Name );
 
-			else result.append_fmt( "struct %s;", Name );
+			else result.append_fmt( "struct %s", Name );
+
+			if ( Parent == nullptr || ( Parent->Type != ECode::Typedef && Parent->Type != ECode::Variable ) )
+				result.append(";");
 		}
 		break;
 
@@ -638,7 +714,7 @@ String AST::to_string()
 
 			if ( Name )
 			{
-				result.append_fmt( "%s\n{\n%s\n};"
+				result.append_fmt( "%s\n{\n%s\n}"
 					, Name
 					, Body->to_string()
 				);
@@ -646,10 +722,13 @@ String AST::to_string()
 			else
 			{
 				// Anonymous union
-				result.append_fmt( "\n{\n%s\n};"
+				result.append_fmt( "\n{\n%s\n}"
 					, Body->to_string()
 				);
 			}
+
+			if ( Parent == nullptr || ( Parent->Type != ECode::Typedef && Parent->Type != ECode::Variable ) )
+				result.append(";");
 		}
 		break;
 
@@ -697,6 +776,9 @@ String AST::to_string()
 				if ( ValueType->ArrExpr )
 					result.append_fmt( "[%s]", ValueType->ArrExpr->to_string() );
 
+				if ( BitfieldSize )
+					result.append_fmt( " : %s", BitfieldSize->to_string() );
+
 				if ( Value )
 					result.append_fmt( " = %s", Value->to_string() );
 
@@ -705,7 +787,10 @@ String AST::to_string()
 				break;
 			}
 
-			if ( UnderlyingType->ArrExpr )
+			if ( BitfieldSize )
+				result.append_fmt( "%s : %s", ValueType->to_string(), BitfieldSize->to_string() );
+
+			else if ( UnderlyingType->ArrExpr )
 				result.append_fmt( "%s %s[%s];", UnderlyingType->to_string(), Name, UnderlyingType->ArrExpr->to_string() );
 
 			else

project/components/data_structures.hpp

@@ -12,6 +12,7 @@ struct AST_Body;
 struct AST_Attributes;
 struct AST_Comment;
 struct AST_Class;
+struct AST_Define;
 struct AST_Enum;
 struct AST_Exec;
 struct AST_Extern;
@@ -23,6 +24,8 @@ struct AST_Namespace;
 struct AST_Operator;
 struct AST_OpCast;
 struct AST_Param;
+struct AST_Pragma;
+struct AST_PreprocessCond;
 struct AST_Specifiers;
 struct AST_Struct;
 struct AST_Template;
@@ -38,6 +41,7 @@ struct CodeBody;
 struct CodeAttributes;
 struct CodeComment;
 struct CodeClass;
+struct CodeDefine;
 struct CodeEnum;
 struct CodeExec;
 struct CodeExtern;
@@ -49,6 +53,8 @@ struct CodeNamespace;
 struct CodeOperator;
 struct CodeOpCast;
 struct CodeParam;
+struct CodePreprocessCond;
+struct CodePragma;
 struct CodeSpecifiers;
 struct CodeStruct;
 struct CodeTemplate;
@@ -115,6 +121,7 @@ struct Code
 	operator CodeAttributes() const;
 	operator CodeComment() const;
 	operator CodeClass() const;
+	operator CodeDefine() const;
 	operator CodeExec() const;
 	operator CodeEnum() const;
 	operator CodeExtern() const;
@@ -126,6 +133,8 @@ struct Code
 	operator CodeOperator() const;
 	operator CodeOpCast() const;
 	operator CodeParam() const;
+	operator CodePragma() const;
+	operator CodePreprocessCond() const;
 	operator CodeSpecifiers() const;
 	operator CodeStruct() const;
 	operator CodeTemplate() const;
@@ -175,6 +184,7 @@ struct AST
 	operator CodeAttributes();
 	operator CodeComment();
 	operator CodeClass();
+	operator CodeDefine();
 	operator CodeEnum();
 	operator CodeExec();
 	operator CodeExtern();
@@ -186,6 +196,8 @@ struct AST
 	operator CodeOperator();
 	operator CodeOpCast();
 	operator CodeParam();
+	operator CodePragma();
+	operator CodePreprocessCond();
 	operator CodeSpecifiers();
 	operator CodeStruct();
 	operator CodeTemplate();
@@ -219,7 +231,10 @@ struct AST
 				AST*  UnderlyingType; // Enum, Typedef
 				AST*  ValueType;      // Parameter, Variable
 			};
+			union {
 				AST*  Params;         // Function, Operator, Template
+				AST*  BitfieldSize;   // Varaiable (Class/Struct Data Member)
+			};
 			union {
 				AST*  ArrExpr;        // Type Symbol
 				AST*  Body;           // Class, Enum, Function, Namespace, Struct, Union
@@ -263,7 +278,10 @@ struct AST_POD
 				AST*  UnderlyingType; // Enum, Typedef
 				AST*  ValueType;      // Parameter, Variable
 			};
+			union {
 				AST*  Params;         // Function, Operator, Template
+				AST*  BitfieldSize;   // Varaiable (Class/Struct Data Member)
+			};
 			union {
 				AST*  ArrExpr;        // Type Symbol
 				AST*  Body;           // Class, Enum, Function, Namespace, Struct, Union
@@ -378,6 +396,7 @@ struct CodeBody
 
 Define_CodeType( Attributes );
 Define_CodeType( Comment );
+Define_CodeType( Define );
 Define_CodeType( Enum );
 Define_CodeType( Exec );
 Define_CodeType( Extern );
@@ -388,6 +407,8 @@ Define_CodeType( Module );
 Define_CodeType( Namespace );
 Define_CodeType( Operator );
 Define_CodeType( OpCast );
+Define_CodeType( Pragma );
+Define_CodeType( PreprocessCond );
 Define_CodeType( Template );
 Define_CodeType( Type );
 Define_CodeType( Typedef );
@@ -474,6 +495,12 @@ struct CodeSpecifiers
 
 	bool append( SpecifierT spec )
 	{
+		if ( ast == nullptr )
+		{
+			log_failure("CodeSpecifiers: Attempted to append to a null specifiers AST!");
+			return false;
+		}
+
 		if ( raw()->NumEntries == AST::ArrSpecs_Cap )
 		{
 			log_failure("CodeSpecifiers: Attempted to append over %d specifiers to a specifiers AST!", AST::ArrSpecs_Cap );
@@ -631,6 +658,21 @@ struct AST_Class
 };
 static_assert( sizeof(AST_Class) == sizeof(AST), "ERROR: AST_Class is not the same size as AST");
 
+struct AST_Define
+{
+	union {
+		char 		  _PAD_[ sizeof(SpecifierT) * AST::ArrSpecs_Cap ];
+		StringCached  Content;
+	};
+	Code              Prev;
+	Code              Next;
+	Code              Parent;
+	StringCached      Name;
+	CodeT             Type;
+	char 			  _PAD_UNUSED_[ sizeof(ModuleFlag) + sizeof(u32) ];
+};
+static_assert( sizeof(AST_Define) == sizeof(AST), "ERROR: AST_Define is not the same size as AST");
+
 struct AST_Enum
 {
 	union {
@@ -847,6 +889,36 @@ struct AST_Param
 };
 static_assert( sizeof(AST_Param) == sizeof(AST), "ERROR: AST_Param is not the same size as AST");
 
+struct AST_Pragma
+{
+	union {
+		char 		  _PAD_[ sizeof(SpecifierT) * AST::ArrSpecs_Cap ];
+		StringCached  Content;
+	};
+	Code              Prev;
+	Code              Next;
+	Code              Parent;
+	StringCached      Name;
+	CodeT             Type;
+	char 			  _PAD_UNUSED_[ sizeof(ModuleFlag) + sizeof(u32) ];
+};
+static_assert( sizeof(AST_Pragma) == sizeof(AST), "ERROR: AST_Pragma is not the same size as AST");
+
+struct AST_PreprocessCond
+{
+	union {
+		char 		  _PAD_[ sizeof(SpecifierT) * AST::ArrSpecs_Cap ];
+		StringCached  Content;
+	};
+	Code              Prev;
+	Code              Next;
+	Code              Parent;
+	StringCached      Name;
+	CodeT             Type;
+	char 			  _PAD_UNUSED_[ sizeof(ModuleFlag) + sizeof(u32) ];
+};
+static_assert( sizeof(AST_PreprocessCond) == sizeof(AST), "ERROR: AST_PreprocessCond is not the same size as AST");
+
 struct AST_Specifiers
 {
 	SpecifierT        ArrSpecs[ AST::ArrSpecs_Cap ];
@@ -998,7 +1070,7 @@ struct AST_Var
 			CodeAttributes Attributes;
 			CodeSpecifiers Specs;
 			CodeType       ValueType;
-			char 	       _PAD_PROPERTIES_[ sizeof(AST*) ];
+			Code           BitfieldSize;
 			Code           Value;
 		};
 	};

project/components/ecode.hpp

@@ -37,7 +37,15 @@ namespace ECode
 	Entry( Operator_Cast )		  \
 	Entry( Operator_Cast_Fwd )    \
 	Entry( Parameters )           \
-	Entry( Preprocessor_Include ) \
+	Entry( Preprocess_Define )    \
+	Entry( Preprocess_If )        \
+	Entry( Preprocess_IfDef )     \
+	Entry( Preprocess_IfNotDef )  \
+	Entry( Preprocess_ElIf )      \
+	Entry( Preprocess_Else )      \
+	Entry( Preprocess_EndIf )     \
+	Entry( Preprocess_Include )   \
+	Entry( Preprocess_Pragma )    \
 	Entry( Specifiers )           \
 	Entry( Struct )               \
 	Entry( Struct_Fwd )           \

project/components/especifier.hpp

@@ -20,6 +20,7 @@ namespace ESpecifier
 	Entry( Internal_Linkage, internal )          \
 	Entry( Local_Persist,    local_persist )     \
 	Entry( Mutable,          mutable )           \
+	Entry( NeverInline,      neverinline )       \
 	Entry( Ptr,              * )                 \
 	Entry( Ref,              & )                 \
 	Entry( Register,         register )          \
@@ -56,9 +57,11 @@ namespace ESpecifier
 		#	pragma push_macro( "global" )
 		#	pragma push_macro( "internal" )
 		#	pragma push_macro( "local_persist" )
+		#	pragma push_macro( "neverinline" )
 		#	undef global
 		#	undef internal
 		#	undef local_persist
+		#	undef neverinline
 
 		#	define Entry( Spec_, Code_ ) { sizeof(stringize(Code_)), stringize(Code_) },
 			Define_Specifiers
@@ -67,6 +70,7 @@ namespace ESpecifier
 		#	pragma pop_macro( "global" )
 		#	pragma pop_macro( "internal" )
 		#	pragma pop_macro( "local_persist" )
+		#	pragma pop_macro( "neverinline" )
 		};
 
 		return lookup[ specifier ];

project/components/etoktype.cpp

@@ -6,7 +6,7 @@ namespace Parser
 
 	For the sake of scanning files, it can scan preprocessor directives
 
-	Attributes_Start is only used to indicate the start of the user_defined attribute list.
+	__Attributes_Start is only used to indicate the start of the user_defined attribute list.
 */
 
 #ifndef GEN_DEFINE_ATTRIBUTE_TOKENS
@@ -33,8 +33,8 @@ namespace Parser
 	Entry( BraceSquare_Close,      "]" )                \
 	Entry( Capture_Start,          "(" )                \
 	Entry( Capture_End,            ")" )                \
-	Entry( Comment,                "comment" )          \
+	Entry( Comment,                "__comment__" )      \
-	Entry( Char,                   "character" )        \
+	Entry( Char,                   "__character__" )    \
 	Entry( Comma,                  "," )                \
 	Entry( Decl_Class,             "class" )            \
 	Entry( Decl_GNU_Attribute,     "__attribute__" )    \
@@ -50,17 +50,23 @@ namespace Parser
 	Entry( Decl_Typedef,           "typedef" )          \
 	Entry( Decl_Using,             "using" )            \
 	Entry( Decl_Union,             "union" )            \
-	Entry( Identifier,             "identifier" )       \
+	Entry( Identifier,             "__identifier__" )   \
 	Entry( Module_Import,          "import" )           \
 	Entry( Module_Export,          "export" )           \
-	Entry( Number,                 "number" )           \
+	Entry( Number,                 "__number__" )       \
-	Entry( Operator,               "operator" )         \
+	Entry( Operator,               "__operator__" )     \
-	Entry( Preprocess_Define,      "#define")           \
+	Entry( Preprocess_Define,      "define")            \
-	Entry( Preproces_Include,      "include" )          \
+	Entry( Preprocess_If,          "if")                \
-	Entry( Preprocess_If,          "#if")               \
+	Entry( Preprocess_IfDef,       "ifdef")             \
-	Entry( Preprocess_Elif,        "#elif")             \
+	Entry( Preprocess_IfNotDef,    "ifndef")            \
-	Entry( Preprocess_Else,        "#else")             \
+	Entry( Preprocess_ElIf,        "elif")              \
-	Entry( Preprocess_EndIf,       "#endif")            \
+	Entry( Preprocess_Else,        "else")              \
+	Entry( Preprocess_EndIf,       "endif")             \
+	Entry( Preprocess_Include,     "include" )          \
+	Entry( Preprocess_Pragma, 	   "pragma")            \
+	Entry( Preprocess_Content, 	   "__macro_content__") \
+	Entry( Preprocess_Macro,       "__macro__")         \
+	Entry( Preprocess_Unsupported, "__unsupported__" )  \
 	Entry( Spec_Alignas,           "alignas" )          \
 	Entry( Spec_Const,             "const" )            \
 	Entry( Spec_Consteval,         "consteval" )        \
@@ -74,13 +80,15 @@ namespace Parser
 	Entry( Spec_Internal_Linkage,  "internal" )         \
 	Entry( Spec_LocalPersist,      "local_persist" )    \
 	Entry( Spec_Mutable,           "mutable" )          \
+	Entry( Spec_NeverInline,       "neverinline" )      \
 	Entry( Spec_Override,          "override" )         \
 	Entry( Spec_Static,            "static" )           \
 	Entry( Spec_ThreadLocal,       "thread_local" )     \
 	Entry( Spec_Volatile,          "volatile")          \
 	Entry( Star,                   "*" )                \
 	Entry( Statement_End,          ";" )                \
-	Entry( String,                 "string" )           \
+	Entry( StaticAssert, 		   "static_assert" )    \
+	Entry( String,                 "__string__" )       \
 	Entry( Type_Unsigned, 	       "unsigned" )         \
 	Entry( Type_Signed,            "signed" )           \
 	Entry( Type_Short,             "short" )            \
@@ -88,8 +96,13 @@ namespace Parser
 	Entry( Type_char, 			   "char" )             \
 	Entry( Type_int, 			   "int" )              \
 	Entry( Type_double, 		   "double" )           \
+	Entry( Type_MS_int8, 		   "__int8" )           \
+	Entry( Type_MS_int16, 		   "__int16" )          \
+	Entry( Type_MS_int32, 		   "__int32" )          \
+	Entry( Type_MS_int64, 		   "__int64" )          \
+	Entry( Type_MS_W64, 		   "_W64" )             \
 	Entry( Varadic_Argument,       "..." )              \
-	Entry( Attributes_Start,       "__attrib_start__" )
+	Entry( __Attributes_Start,     "__attrib_start__" )
 
 	namespace ETokType
 	{

project/components/header_end.hpp

@@ -26,12 +26,29 @@ void AST::append( AST* other )
 
 char const* AST::debug_str()
 {
+	if ( Parent )
+	{
+		char const* fmt = stringize(
+			\nType    : %s
+			\nParent  : %s %s
+			\nName    : %s
+		);
+
+		// These should be used immediately in a log.
+		// Thus if its desired to keep the debug str
+		// for multiple calls to bprintf,
+		// allocate this to proper string.
+		return str_fmt_buf( fmt
+			,	type_str()
+			,	Parent->Name
+			,   Parent->type_str()
+			,	Name     ? Name         : ""
+		);
+	}
+
 	char const* fmt = stringize(
-		\nCode Debug:
 		\nType    : %s
-		\nParent  : %s
 		\nName    : %s
-		\nComment : %s
 	);
 
 	// These should be used immediately in a log.
@@ -40,7 +57,6 @@ char const* AST::debug_str()
 	// allocate this to proper string.
 	return str_fmt_buf( fmt
 		,	type_str()
-		,	Parent   ? Parent->Name : ""
 		,	Name     ? Name         : ""
 	);
 }
@@ -162,6 +178,7 @@ Define_CodeImpl( CodeBody );
 Define_CodeImpl( CodeAttributes );
 Define_CodeImpl( CodeComment );
 Define_CodeImpl( CodeClass );
+Define_CodeImpl( CodeDefine );
 Define_CodeImpl( CodeEnum );
 Define_CodeImpl( CodeExec );
 Define_CodeImpl( CodeExtern );
@@ -173,6 +190,8 @@ Define_CodeImpl( CodeNamespace );
 Define_CodeImpl( CodeOperator );
 Define_CodeImpl( CodeOpCast );
 Define_CodeImpl( CodeParam );
+Define_CodeImpl( CodePragma );
+Define_CodeImpl( CodePreprocessCond );
 Define_CodeImpl( CodeSpecifiers );
 Define_CodeImpl( CodeStruct );
 Define_CodeImpl( CodeTemplate );
@@ -193,6 +212,7 @@ Define_AST_Cast( Body );
 Define_AST_Cast( Attributes );
 Define_AST_Cast( Comment );
 Define_AST_Cast( Class );
+Define_AST_Cast( Define );
 Define_AST_Cast( Enum );
 Define_AST_Cast( Exec );
 Define_AST_Cast( Extern );
@@ -204,6 +224,8 @@ Define_AST_Cast( Namespace );
 Define_AST_Cast( Operator );
 Define_AST_Cast( OpCast );
 Define_AST_Cast( Param );
+Define_AST_Cast( Pragma );
+Define_AST_Cast( PreprocessCond );
 Define_AST_Cast( Struct );
 Define_AST_Cast( Specifiers );
 Define_AST_Cast( Template );
@@ -223,6 +245,7 @@ Code::operator Code ## type() const \
 Define_CodeCast( Attributes );
 Define_CodeCast( Comment );
 Define_CodeCast( Class );
+Define_CodeCast( Define );
 Define_CodeCast( Exec );
 Define_CodeCast( Enum );
 Define_CodeCast( Extern );
@@ -234,6 +257,8 @@ Define_CodeCast( Namespace );
 Define_CodeCast( Operator );
 Define_CodeCast( OpCast );
 Define_CodeCast( Param );
+Define_CodeCast( Pragma );
+Define_CodeCast( PreprocessCond );
 Define_CodeCast( Specifiers );
 Define_CodeCast( Struct );
 Define_CodeCast( Template );
@@ -368,28 +393,6 @@ StrC token_fmt_impl( sw num, ... )
 
 #pragma region Constants
 
-#ifdef GEN_DEFINE_LIBRARY_CODE_CONSTANTS
-	// Predefined typename codes. Are set to readonly and are setup during gen::init()
-
-	extern CodeType t_b32;
-
-	extern CodeType t_s8;
-	extern CodeType t_s16;
-	extern CodeType t_s32;
-	extern CodeType t_s64;
-
-	extern CodeType t_u8;
-	extern CodeType t_u16;
-	extern CodeType t_u32;
-	extern CodeType t_u64;
-
-	extern CodeType t_sw;
-	extern CodeType t_uw;
-
-	extern CodeType t_f32;
-	extern CodeType t_f64;
-#endif
-
 #ifndef GEN_GLOBAL_BUCKET_SIZE
 #	define GEN_GLOBAL_BUCKET_SIZE megabytes(10)
 #endif
@@ -437,29 +440,22 @@ constexpr s32 TokenFmt_TokenMap_MemSize	= GEN_TOKEN_FMT_TOKEN_MAP_MEM_SIZE;
 constexpr s32 LexAllocator_Size         = GEN_LEX_ALLOCATOR_SIZE;
 constexpr s32 Builder_StrBufferReserve  = GEN_BUILDER_STR_BUFFER_RESERVE;
 
-extern CodeType t_empty; // Used with varaidc parameters. (Exposing just in case its useful for another circumstance)
-extern CodeType t_auto;
-extern CodeType t_void;
-extern CodeType t_int;
-extern CodeType t_bool;
-extern CodeType t_char;
-extern CodeType t_wchar_t;
-extern CodeType t_class;
-extern CodeType t_typename;
-
-extern CodeParam param_varadic;
-
-extern CodeAttributes attrib_api_export;
-extern CodeAttributes attrib_api_import;
-
 extern Code access_public;
 extern Code access_protected;
 extern Code access_private;
 
+extern CodeAttributes attrib_api_export;
+extern CodeAttributes attrib_api_import;
+
 extern Code module_global_fragment;
 extern Code module_private_fragment;
 
-extern Code pragma_once;
+extern CodePragma pragma_once;
+
+extern CodeParam param_varadic;
+
+extern CodePreprocessCond preprocess_else;
+extern CodePreprocessCond preprocess_endif;
 
 extern CodeSpecifiers spec_const;
 extern CodeSpecifiers spec_consteval;
@@ -472,6 +468,7 @@ extern CodeSpecifiers spec_inline;
 extern CodeSpecifiers spec_internal_linkage;
 extern CodeSpecifiers spec_local_persist;
 extern CodeSpecifiers spec_mutable;
+extern CodeSpecifiers spec_neverinline;
 extern CodeSpecifiers spec_override;
 extern CodeSpecifiers spec_ptr;
 extern CodeSpecifiers spec_ref;
@@ -482,6 +479,38 @@ extern CodeSpecifiers spec_thread_local;
 extern CodeSpecifiers spec_virtual;
 extern CodeSpecifiers spec_volatile;
 
+extern CodeType t_empty; // Used with varaidc parameters. (Exposing just in case its useful for another circumstance)
+extern CodeType t_auto;
+extern CodeType t_void;
+extern CodeType t_int;
+extern CodeType t_bool;
+extern CodeType t_char;
+extern CodeType t_wchar_t;
+extern CodeType t_class;
+extern CodeType t_typename;
+
+#ifdef GEN_DEFINE_LIBRARY_CODE_CONSTANTS
+	// Predefined typename codes. Are set to readonly and are setup during gen::init()
+
+	extern CodeType t_b32;
+
+	extern CodeType t_s8;
+	extern CodeType t_s16;
+	extern CodeType t_s32;
+	extern CodeType t_s64;
+
+	extern CodeType t_u8;
+	extern CodeType t_u16;
+	extern CodeType t_u32;
+	extern CodeType t_u64;
+
+	extern CodeType t_sw;
+	extern CodeType t_uw;
+
+	extern CodeType t_f32;
+	extern CodeType t_f64;
+#endif
+
 #pragma endregion Constants
 
 #pragma region Macros

project/components/interface.cpp

@@ -1,3 +1,6 @@
+internal void init_parser();
+internal void deinit_parser();
+
 internal
 void* Global_Allocator_Proc( void* allocator_data, AllocType type, sw size, sw alignment, void* old_memory, sw old_size, u64 flags )
 {
@@ -71,6 +74,64 @@ void define_constants()
 	Code::Invalid = make_code();
 	Code::Invalid.set_global();
 
+	t_empty          = (CodeType) make_code();
+	t_empty->Type    = ECode::Typename;
+	t_empty->Name    = get_cached_string( txt_StrC("") );
+	t_empty.set_global();
+
+	access_private       = make_code();
+	access_private->Type = ECode::Access_Private;
+	access_private->Name = get_cached_string( txt_StrC("private:") );
+	access_private.set_global();
+
+	access_protected       = make_code();
+	access_protected->Type = ECode::Access_Protected;
+	access_protected->Name = get_cached_string( txt_StrC("protected:") );
+	access_protected.set_global();
+
+	access_public       = make_code();
+	access_public->Type = ECode::Access_Public;
+	access_public->Name = get_cached_string( txt_StrC("public:") );
+	access_public.set_global();
+
+	attrib_api_export = def_attributes( code(GEN_API_Export_Code));
+	attrib_api_export.set_global();
+
+	attrib_api_import = def_attributes( code(GEN_API_Import_Code));
+	attrib_api_import.set_global();
+
+	module_global_fragment          = make_code();
+	module_global_fragment->Type    = ECode::Untyped;
+	module_global_fragment->Name    = get_cached_string( txt_StrC("module;") );
+	module_global_fragment->Content = module_global_fragment->Name;
+	module_global_fragment.set_global();
+
+	module_private_fragment          = make_code();
+	module_private_fragment->Type    = ECode::Untyped;
+	module_private_fragment->Name    = get_cached_string( txt_StrC("module : private;") );
+	module_private_fragment->Content = module_private_fragment->Name;
+	module_private_fragment.set_global();
+
+	pragma_once          = (CodePragma) make_code();
+	pragma_once->Type    = ECode::Untyped;
+	pragma_once->Name    = get_cached_string( txt_StrC("once") );
+	pragma_once->Content = pragma_once->Name;
+	pragma_once.set_global();
+
+	param_varadic            = (CodeType) make_code();
+	param_varadic->Type      = ECode::Parameters;
+	param_varadic->Name      = get_cached_string( txt_StrC("...") );
+	param_varadic->ValueType = t_empty;
+	param_varadic.set_global();
+
+	preprocess_else = (CodePreprocessCond) make_code();
+	preprocess_else->Type = ECode::Preprocess_Else;
+	preprocess_else.set_global();
+
+	preprocess_endif = (CodePreprocessCond) make_code();
+	preprocess_endif->Type = ECode::Preprocess_EndIf;
+	preprocess_endif.set_global();
+
 #	define def_constant_code_type( Type_ )   \
 		t_##Type_ = def_type( name(Type_) ); \
 		t_##Type_.set_global();
@@ -105,62 +166,14 @@ void define_constants()
 #endif
 #	undef def_constant_code_type
 
-	t_empty          = (CodeType) make_code();
-	t_empty->Type    = ECode::Typename;
-	t_empty->Name    = get_cached_string( txt_StrC("") );
-	t_empty.set_global();
-
-	param_varadic            = (CodeType) make_code();
-	param_varadic->Type      = ECode::Parameters;
-	param_varadic->Name      = get_cached_string( txt_StrC("...") );
-	param_varadic->ValueType = t_empty;
-	param_varadic.set_global();
-
-	attrib_api_export = def_attributes( code(GEN_API_Export_Code));
-	attrib_api_export.set_global();
-
-	attrib_api_import = def_attributes( code(GEN_API_Import_Code));
-	attrib_api_import.set_global();
-
-	access_private       = make_code();
-	access_private->Type = ECode::Access_Private;
-	access_private->Name = get_cached_string( txt_StrC("private:") );
-	access_private.set_global();
-
-	access_protected       = make_code();
-	access_protected->Type = ECode::Access_Protected;
-	access_protected->Name = get_cached_string( txt_StrC("protected:") );
-	access_protected.set_global();
-
-	access_public       = make_code();
-	access_public->Type = ECode::Access_Public;
-	access_public->Name = get_cached_string( txt_StrC("public:") );
-	access_public.set_global();
-
-	module_global_fragment          = make_code();
-	module_global_fragment->Type    = ECode::Untyped;
-	module_global_fragment->Name    = get_cached_string( txt_StrC("module;") );
-	module_global_fragment->Content = module_global_fragment->Name;
-	module_global_fragment.set_global();
-
-	module_private_fragment          = make_code();
-	module_private_fragment->Type    = ECode::Untyped;
-	module_private_fragment->Name    = get_cached_string( txt_StrC("module : private;") );
-	module_private_fragment->Content = module_private_fragment->Name;
-	module_private_fragment.set_global();
-
-	pragma_once          = make_code();
-	pragma_once->Type    = ECode::Untyped;
-	pragma_once->Name    = get_cached_string( txt_StrC("#pragma once") );
-	pragma_once->Content = pragma_once->Name;
-	pragma_once.set_global();
-
 #	pragma push_macro( "global" )
 #	pragma push_macro( "internal" )
 #	pragma push_macro( "local_persist" )
+#	pragma push_macro( "neverinline" )
 #	undef global
 #	undef internal
 #	undef local_persist
+#	undef neverinline
 
 #	define def_constant_spec( Type_, ... )                                  \
 		spec_##Type_ = def_specifiers( num_args(__VA_ARGS__), __VA_ARGS__); \
@@ -177,6 +190,7 @@ void define_constants()
 	def_constant_spec( internal_linkage, ESpecifier::Internal_Linkage );
 	def_constant_spec( local_persist,    ESpecifier::Local_Persist );
 	def_constant_spec( mutable,          ESpecifier::Mutable );
+	def_constant_spec( neverinline,      ESpecifier::NeverInline );
 	def_constant_spec( override,         ESpecifier::Override );
 	def_constant_spec( ptr,              ESpecifier::Ptr );
 	def_constant_spec( ref,              ESpecifier::Ref );
@@ -193,6 +207,7 @@ void define_constants()
 #	pragma pop_macro( "global" )
 #	pragma pop_macro( "internal" )
 #	pragma pop_macro( "local_persist" )
+#	pragma pop_macro( "neverinline" )
 
 #	undef def_constant_spec
 }
@@ -258,6 +273,7 @@ void init()
 	}
 
 	define_constants();
+	init_parser();
 }
 
 void deinit()
@@ -300,6 +316,7 @@ void deinit()
 	while ( left--, left );
 
 	Global_AllocatorBuckets.free();
+	deinit_parser();
 }
 
 void reset()

project/components/interface.hpp

@@ -45,6 +45,8 @@ CodeClass def_class( StrC name
 	, ModuleFlag     mflags       = ModuleFlag::None
 	, CodeType*      interfaces   = nullptr, s32 num_interfaces = 0 );
 
+CodeDefine def_define( StrC name, StrC content );
+
 CodeEnum def_enum( StrC name
 	, Code         body      = NoCode,      CodeType       type       = NoCode
 	, EnumT        specifier = EnumRegular, CodeAttributes attributes = NoCode
@@ -63,7 +65,7 @@ CodeInclude   def_include  ( StrC content );
 CodeModule    def_module   ( StrC name,            ModuleFlag mflags = ModuleFlag::None );
 CodeNamespace def_namespace( StrC name, Code body, ModuleFlag mflags = ModuleFlag::None );
 
-CodeOperator def_operator( OperatorT op
+CodeOperator def_operator( OperatorT op, StrC nspace
 	, CodeParam      params     = NoCode, CodeType       ret_type   = NoCode, Code body = NoCode
 	, CodeSpecifiers specifiers = NoCode, CodeAttributes attributes = NoCode
 	, ModuleFlag     mflags     = ModuleFlag::None );
@@ -71,6 +73,10 @@ CodeOperator def_operator( OperatorT op
 CodeOpCast def_operator_cast( CodeType type, Code body = NoCode, CodeSpecifiers specs = NoCode );
 
 CodeParam  def_param ( CodeType type, StrC name, Code value = NoCode );
+CodePragma def_pragma( StrC directive );
+
+CodePreprocessCond def_preprocess_cond( EPreprocessCond type, StrC content );
+
 CodeSpecifiers def_specifier( SpecifierT specifier );
 
 CodeStruct def_struct( StrC name

project/components/interface.upfront.cpp

@@ -489,6 +489,26 @@ CodeClass def_class( StrC name
 	return result;
 }
 
+CodeDefine def_define( StrC name, StrC content )
+{
+	using namespace ECode;
+
+	name_check( def_define, name );
+
+	if ( content.Len <= 0 || content.Ptr == nullptr )
+	{
+		log_failure( "gen::def_define: Invalid value provided" );
+		return CodeInvalid;
+	}
+
+	CodeDefine
+	result          = (CodeDefine) make_code();
+	result->Name    = get_cached_string( name );
+	result->Content = get_cached_string( content );
+
+	return result;
+}
+
 CodeEnum def_enum( StrC name
 	, Code       body,      CodeType       type
 	, EnumT      specifier, CodeAttributes attributes
@@ -719,7 +739,7 @@ CodeInclude def_include ( StrC path )
 
 	Code
 	result          = make_code();
-	result->Type    = ECode::Preprocessor_Include;
+	result->Type    = ECode::Preprocess_Include;
 	result->Name    = get_cached_string( path );
 	result->Content = result->Name;
 
@@ -763,7 +783,7 @@ CodeNamespace def_namespace( StrC name, Code body, ModuleFlag mflags )
 	return result;
 }
 
-CodeOperator def_operator( OperatorT op
+CodeOperator def_operator( OperatorT op, StrC nspace
 	, CodeParam      params_code, CodeType       ret_type, Code body
 	, CodeSpecifiers specifiers,  CodeAttributes attributes
 	, ModuleFlag     mflags )
@@ -789,7 +809,7 @@ CodeOperator def_operator( OperatorT op
 		return CodeInvalid;
 	}
 
-	char const* name = str_fmt_buf( "operator %s", to_str(op) );
+	char const* name = str_fmt_buf( "%.*soperator %s", nspace.Len, nspace.Ptr, to_str(op) );
 
 	CodeOperator
 	result              = (CodeOperator) make_code();
@@ -910,6 +930,53 @@ CodeParam def_param( CodeType type, StrC name, Code value )
 	return result;
 }
 
+CodePragma def_pragma( StrC directive )
+{
+	using namespace ECode;
+
+	if ( directive.Len <= 0 || directive.Ptr == nullptr )
+	{
+		log_failure( "gen::def_comment: Invalid comment provided:" );
+		return CodeInvalid;
+	}
+
+	CodePragma
+	result          = (CodePragma) make_code();
+	result->Type    = Preprocess_Pragma;
+	result->Content = get_cached_string( directive );
+
+	return result;
+}
+
+CodePreprocessCond def_preprocess_cond( EPreprocessCond type, StrC expr )
+{
+	using namespace ECode;
+
+	if ( expr.Len <= 0 || expr.Ptr == nullptr )
+	{
+		log_failure( "gen::def_comment: Invalid comment provided:" );
+		return CodeInvalid;
+	}
+
+	CodePreprocessCond
+	result          = (CodePreprocessCond) make_code();
+	result->Content = get_cached_string( expr );
+
+	switch (type)
+	{
+		case EPreprocessCond::If:
+			result->Type = ECode::Preprocess_If;
+		case EPreprocessCond::IfDef:
+			result->Type = Preprocess_IfDef;
+		case EPreprocessCond::IfNotDef:
+			result->Type = Preprocess_IfNotDef;
+		case EPreprocessCond::ElIf:
+			result->Type = Preprocess_ElIf;
+	}
+
+	return result;
+}
+
 CodeSpecifiers def_specifier( SpecifierT spec )
 {
 	CodeSpecifiers

project/components/static_data.cpp

@@ -22,6 +22,45 @@ global AllocatorInfo Allocator_TypeTable   = heap();
 
 #pragma region Constants
 
+global Code access_public;
+global Code access_protected;
+global Code access_private;
+
+global CodeAttributes attrib_api_export;
+global CodeAttributes attrib_api_import;
+
+global Code module_global_fragment;
+global Code module_private_fragment;
+
+global CodeParam param_varadic;
+
+global CodePragma pragma_once;
+
+global CodePreprocessCond preprocess_else;
+global CodePreprocessCond preprocess_endif;
+
+global CodeSpecifiers spec_const;
+global CodeSpecifiers spec_consteval;
+global CodeSpecifiers spec_constexpr;
+global CodeSpecifiers spec_constinit;
+global CodeSpecifiers spec_extern_linkage;
+global CodeSpecifiers spec_final;
+global CodeSpecifiers spec_global;
+global CodeSpecifiers spec_inline;
+global CodeSpecifiers spec_internal_linkage;
+global CodeSpecifiers spec_local_persist;
+global CodeSpecifiers spec_mutable;
+global CodeSpecifiers spec_neverinline;
+global CodeSpecifiers spec_override;
+global CodeSpecifiers spec_ptr;
+global CodeSpecifiers spec_ref;
+global CodeSpecifiers spec_register;
+global CodeSpecifiers spec_rvalue;
+global CodeSpecifiers spec_static_member;
+global CodeSpecifiers spec_thread_local;
+global CodeSpecifiers spec_virtual;
+global CodeSpecifiers spec_volatile;
+
 global CodeType t_empty;
 global CodeType t_auto;
 global CodeType t_void;
@@ -52,39 +91,4 @@ global CodeType t_f32;
 global CodeType t_f64;
 #endif
 
-global CodeParam param_varadic;
-
-global CodeAttributes attrib_api_export;
-global CodeAttributes attrib_api_import;
-
-global Code access_public;
-global Code access_protected;
-global Code access_private;
-
-global Code module_global_fragment;
-global Code module_private_fragment;
-
-global Code pragma_once;
-
-global CodeSpecifiers spec_const;
-global CodeSpecifiers spec_consteval;
-global CodeSpecifiers spec_constexpr;
-global CodeSpecifiers spec_constinit;
-global CodeSpecifiers spec_extern_linkage;
-global CodeSpecifiers spec_final;
-global CodeSpecifiers spec_global;
-global CodeSpecifiers spec_inline;
-global CodeSpecifiers spec_internal_linkage;
-global CodeSpecifiers spec_local_persist;
-global CodeSpecifiers spec_mutable;
-global CodeSpecifiers spec_override;
-global CodeSpecifiers spec_ptr;
-global CodeSpecifiers spec_ref;
-global CodeSpecifiers spec_register;
-global CodeSpecifiers spec_rvalue;
-global CodeSpecifiers spec_static_member;
-global CodeSpecifiers spec_thread_local;
-global CodeSpecifiers spec_virtual;
-global CodeSpecifiers spec_volatile;
-
 #pragma endregion Constants

project/components/types.hpp

@@ -8,16 +8,6 @@ using LogFailType = sw(*)(char const*, ...);
 	constexpr LogFailType log_failure = fatal;
 #endif
 
-// Used to indicate if enum definitoin is an enum class or regular enum.
-enum class EnumT : u8
-{
-	Regular,
-	Class
-};
-
-constexpr EnumT EnumClass   = EnumT::Class;
-constexpr EnumT EnumRegular = EnumT::Regular;
-
 enum class AccessSpec : u32
 {
 	Default,
@@ -46,6 +36,17 @@ char const* to_str( AccessSpec type )
 	return lookup[ (u32)type ];
 }
 
+// Used to indicate if enum definitoin is an enum class or regular enum.
+enum class EnumT : u8
+{
+	Regular,
+	Class
+};
+
+constexpr EnumT EnumClass   = EnumT::Class;
+constexpr EnumT EnumRegular = EnumT::Regular;
+
+
 enum class ModuleFlag : u32
 {
 	None    = 0,
@@ -62,6 +63,19 @@ ModuleFlag operator|( ModuleFlag A, ModuleFlag B)
 	return (ModuleFlag)( (u32)A | (u32)B );
 }
 
+enum class EPreprocessCond : u32
+{
+	If,
+	IfDef,
+	IfNotDef,
+	ElIf
+};
+
+constexpr EPreprocessCond PreprocessCond_If       = EPreprocessCond::If;
+constexpr EPreprocessCond PreprocessCond_IfDef    = EPreprocessCond::IfDef;
+constexpr EPreprocessCond PreprocessCond_IfNotDef = EPreprocessCond::IfNotDef;
+constexpr EPreprocessCond PreprocessCond_ElIf     = EPreprocessCond::ElIf;
+
 /*
 	Predefined attributes
 	Used for the parser constructors to identify non-standard attributes

project/components/untyped.cpp

@@ -99,17 +99,35 @@ sw token_fmt_va( char* buf, uw buf_size, s32 num_tokens, va_list va )
 
 Code untyped_str( StrC content )
 {
+	if ( content.Len == 0 )
+	{
+		log_failure( "untyped_str: empty string" );
+		return CodeInvalid;
+	}
+
 	Code
 	result          = make_code();
 	result->Name    = get_cached_string( content );
 	result->Type    = ECode::Untyped;
 	result->Content = result->Name;
 
+	if ( result->Name == nullptr )
+	{
+		log_failure( "untyped_str: could not cache string" );
+		return CodeInvalid;
+	}
+
 	return result;
 }
 
 Code untyped_fmt( char const* fmt, ...)
 {
+	if ( fmt == nullptr )
+	{
+		log_failure( "untyped_fmt: null format string" );
+		return CodeInvalid;
+	}
+
 	local_persist thread_local
 	char buf[GEN_PRINTF_MAXLEN] = { 0 };
 
@@ -124,11 +142,23 @@ Code untyped_fmt( char const* fmt, ...)
 	result->Type    = ECode::Untyped;
 	result->Content = get_cached_string( { length, buf } );
 
+	if ( result->Name == nullptr )
+	{
+		log_failure( "untyped_fmt: could not cache string" );
+		return CodeInvalid;
+	}
+
 	return result;
 }
 
 Code untyped_token_fmt( s32 num_tokens, ... )
 {
+	if ( num_tokens == 0 )
+	{
+		log_failure( "untyped_token_fmt: zero tokens" );
+		return CodeInvalid;
+	}
+
 	local_persist thread_local
 	char buf[GEN_PRINTF_MAXLEN] = { 0 };
 
@@ -143,5 +173,11 @@ Code untyped_token_fmt( s32 num_tokens, ... )
 	result->Type    = ECode::Untyped;
 	result->Content = result->Name;
 
+	if ( result->Name == nullptr )
+	{
+		log_failure( "untyped_fmt: could not cache string" );
+		return CodeInvalid;
+	}
+
 	return result;
 }

project/dependencies/file_handling.cpp

@@ -224,7 +224,7 @@ internal GEN_FILE_CLOSE_PROC( _posix_file_close )
 
 FileOperations const default_file_operations = { _posix_file_read, _posix_file_write, _posix_file_seek, _posix_file_close };
 
-GEN_NEVER_INLINE GEN_FILE_OPEN_PROC( _posix_file_open )
+neverinline GEN_FILE_OPEN_PROC( _posix_file_open )
 {
 	s32 os_mode;
 	switch ( mode & GEN_FILE_MODES )

project/dependencies/macros.hpp

@@ -86,6 +86,9 @@
 	}                      \
 	while(0);
 
+#define labeled_scope_start if ( false ) {
+#define labeled_scope_end   }
+
 #define clamp( x, lower, upper )      min( max( ( x ), ( lower ) ), ( upper ) )
 #define count_of( x )                 ( ( size_of( x ) / size_of( 0 [ x ] ) ) / ( ( sw )( ! ( size_of( x ) % size_of( 0 [ x ] ) ) ) ) )
 #define is_between( x, lower, upper ) ( ( ( lower ) <= ( x ) ) && ( ( x ) <= ( upper ) ) )

project/enums/ECode.csv

@@ -32,7 +32,15 @@ Operator_Member_Fwd
 Operator_Cast
 Operator_Cast_Fwd
 Parameters
-Preprocessor_Include
+Preprocess_Define
+Preprocess_Include
+Preprocess_If
+Preprocess_IfDef
+Preprocess_IfNotDef
+Preprocess_ElIf
+Preprocess_Else
+Preprocess_EndIf
+Preprocess_Pragma
 Specifiers
 Struct
 Struct_Fwd

project/enums/ESpecifier.csv

@@ -9,6 +9,7 @@ Inline,           inline
 Internal_Linkage, internal
 Local_Persist,    local_persist
 Mutable,          mutable
+NeverInline,      neverinline
 Ptr,              *
 Ref,              &
 Register,         register

project/enums/ETokType.csv

@@ -15,8 +15,8 @@ BraceSquare_Open,       "["
 BraceSquare_Close,      "]"
 Capture_Start,          "("
 Capture_End,            ")"
-Comment,                "comemnt"
+Comment,                "__comemnt__"
-Char,                   "character"
+Char,                   "__character__"
 Comma,                  ","
 Decl_Class,             "class"
 Decl_GNU_Attribute,     "__attribute__"
@@ -26,23 +26,28 @@ Decl_Extern_Linkage,    "extern"
 Decl_Friend,            "friend"
 Decl_Module,            "module"
 Decl_Namespace,         "namespace"
-Decl_Operator,          "operator"
+Decl_Operator,          "__operator__"
 Decl_Struct,            "struct"
 Decl_Template,          "template"
 Decl_Typedef,           "typedef"
 Decl_Using,             "using"
 Decl_Union,             "union"
-Identifier,             "identifier"
+Identifier,             "__identifier__"
 Module_Import,          "import"
 Module_Export,          "export"
-Number,                 "number"
+Number,                 "__number__"
-Operator,               "operator"
+Operator,               "__operator__"
-Preprocess_Define,      "#define"
+Preprocess_Define,      "define"
-Preprocess_Include,     "#include"
+Preprocess_If,          "if"
-Preprocess_If,          "#if"
+Preprocess_IfDef, 	    "ifdef"
-Preprocess_ElIF,		"#elif"
+Preprocess_IfNotDef, 	"ifndef"
-Preprocess_Else,        "#else"
+Preprocess_ElIf,		"elif"
-Preprocess_EndIf,       "#endif"
+Preprocess_Else,        "else"
+Preprocess_EndIf,       "endif"
+Preprocess_Include,     "include"
+Preprocess_Pragma,      "pragma"
+Preprocess_Macro,       "__macro__"
+Preprocess_Unsupported, "__unsupported__"
 Spec_Alignas,           "alignas"
 Spec_Const,             "const"
 Spec_Consteval,         "consteval"
@@ -56,13 +61,15 @@ Spec_Inline,            "inline"
 Spec_Internal_Linkage,  "internal"
 Spec_LocalPersist,      "local_persist"
 Spec_Mutable,           "mutable"
+Spec_NeverInline,       "neverinline"
 Spec_Override,          "override"
 Spec_Static,            "static"
 Spec_ThreadLocal,       "thread_local"
 Spec_Volatile,          "volatile"
 Star,                   "*"
 Statement_End,          ";"
-String,                 "string"
+StaticAssert,           "static_assert"
+String,                 "__string__"
 Type_Unsigned,          "unsigned"
 Type_Signed,            "signed"
 Type_Short,             "short"
@@ -70,5 +77,10 @@ Type_Long,              "long"
 Type_char,              "char"
 Type_int,               "int"
 Type_double,            "double"
+Type_MS_int8, 		    "__int8"
+Type_MS_int16, 		    "__int16"
+Type_MS_int32, 		    "__int32"
+Type_MS_int64, 		    "__int64"
+Type_MS_W64, 		    "_W64"
 Varadic_Argument,       "..."
-Attributes_Start,       "__attrib_start__"
+__Attributes_Start,     "__attrib_start__"

project/file_processors/builder.cpp

@@ -1,6 +1,6 @@
 void Builder::print( Code code )
 {
-	Buffer.append_fmt( "%s\n", code->to_string() );
+	Buffer.append( code->to_string() );
 }
 
 void Builder::print_fmt( char const* fmt, ... )

scripts/.clang-format

@@ -73,7 +73,7 @@ BreakBeforeTernaryOperators: true
 BreakConstructorInitializers: BeforeComma
 BreakStringLiterals: true
 
-ColumnLimit: 180
+ColumnLimit: 120
 
 CompactNamespaces: true
 
@@ -94,16 +94,16 @@ IncludeBlocks: Preserve
 
 
 IndentCaseBlocks: false
-IndentCaseLabels: false
+IndentCaseLabels: true
 IndentExternBlock: AfterExternBlock
 IndentGotoLabels: true
 IndentPPDirectives: AfterHash
 IndentRequires: true
 IndentWidth: 4
-IndentWrappedFunctionNames: false
+IndentWrappedFunctionNames: true
 
 # InsertNewlineAtEOF: true
-InsertTrailingCommas: Wrapped
+# InsertTrailingCommas: Wrapped
 
 LambdaBodyIndentation: OuterScope
 

scripts/msvc/devshell.ps1

@@ -1,26 +1,23 @@
-# This script is used to iniitate the MSVC DevShell
+$ErrorActionPreference = "Stop"
-$vs_devshell = @()
-@("enterprise", "professional", "community") | ForEach-Object {
-    $vs_devshell_2022 = "C:\Program Files\Microsoft Visual Studio\2022\"       + $_ + "\Common7\Tools\Launch-VsDevShell.ps1"
-    $vs_devshell_2019 = "C:\Program Files (x86)\Microsoft Visual Studio\2019\" + $_ + "\Common7\Tools\Launch-VsDevShell.ps1"
 
-    $vs_devshell += @( $vs_devshell_2022, $vs_devshell_2019 )
+# Use vswhere to find the latest Visual Studio installation
+$vswhere_out = & "C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe" -latest -property installationPath
+if ($null -eq $vswhere_out) {
+    Write-Host "ERROR: Visual Studio installation not found"
+    exit 1
 }
 
-$found = $false
+# Find Launch-VsDevShell.ps1 in the Visual Studio installation
-foreach($path in $vs_devshell) {
+$vs_path     = $vswhere_out
-    if (Test-Path $path) {
+$vs_devshell = Join-Path $vs_path "\Common7\Tools\Launch-VsDevShell.ps1"
-        write-host "Found $path"
 
-        Push-Location # Save the current path, loading the script will change it.
+if ( -not (Test-Path $vs_devshell) ) {
-        & $path
+    Write-Host "ERROR: Launch-VsDevShell.ps1 not found in Visual Studio installation"
-        Pop-Location
+    Write-Host Tested path: $vs_devshell
-
+    exit 1
-        $found = $true
-        break;
-    }
 }
 
-if (-not $found) {
+# Launch the Visual Studio Developer Shell
-    write-host "MSVC DevShell: No valid path found"
+Push-Location
-}
+& $vs_devshell @args
+Pop-Location

test/test.cpp

@@ -5,6 +5,7 @@
 #include "gen.cpp"
 #include "sanity.cpp"
 #include "SOA.cpp"
+#include "test.singleheader_ast.cpp"
 
 #if GEN_TIME
 int gen_main()
@@ -12,10 +13,12 @@ int gen_main()
 	using namespace gen;
 	log_fmt("\ngen_time:");
 
-	check_sanity();
+	// check_sanity();
 
 	check_SOA();
 
+	check_singleheader_ast();
+
 	return 0;
 }
 #endif

test/test.singleheader_ast.cpp

@@ -0,0 +1,42 @@
+#pragma once
+
+#define GEN_DEFINE_LIBRARY_CODE_CONSTANTS
+#define GEN_ENFORCE_STRONG_CODE_TYPES
+#define GEN_EXPOSE_BACKEND
+#define GEN_BENCHMARK
+#include "gen.hpp"
+#include "file_processors/scanner.hpp"
+using namespace gen;
+
+void check_singleheader_ast()
+{
+	#define project_dir "../../"
+	gen::init();
+	log_fmt("\ncheck_singleheader_ast:\n");
+
+	FileContents file = file_read_contents( GlobalAllocator, true, project_dir "singleheader/gen/gen.hpp" );
+
+	CodeBody ast = parse_global_body( { file.size, (char const*)file.data } );
+
+	log_fmt("generated AST!!!\n");
+
+	s32 idx = 0;
+	for ( Code entry : ast )
+	{
+		if (idx == 900)
+		{
+			log_fmt("break here\n");
+		}
+		log_fmt("Entry %d: %s\n", idx, entry.to_string() );
+		idx++;
+	}
+
+	Builder builder;
+	builder.open( "singleheader_copy.gen.hpp" );
+	log_fmt("\n\nserializng ast\n");
+	builder.print( ast );
+	builder.write();
+
+	log_fmt("passed!!\n");
+	gen::deinit();
+}