Large updates to docs

base/Readme.md

@@ -1,4 +1,10 @@
-# Documentation
+## Navigation
+
+# base
+
+[Top](../Readme.md)
+
+* [docs](../docs/Readme.md)
 
 The library is fragmented into a series of headers and source files meant to be scanned in and then generated to a standard target format, or a user's desires.
 
@@ -17,17 +23,108 @@ Standard formats:
     * **`Scanner`**: Interface to load up `Code` from files two basic funcctions are currently provided.
       * `scan_file`: Used mainly by the library format generators to directly scan files into untyped `Code` (raw string content, pre-formatted no AST parsed).
       * `parse_file`: Used to read file and then parsed to populate a `CodeBody` AST.
+      * CSV parsing via one or two columns simplified.
 * **gen_segemetned**: Dependencies go into gen.dep.{hpp/cpp} and components into gen.{hpp/cpp}
 * **gen_singleheader**: Everything into a single file: gen.hpp
 * **gen_unreal_engine**: Like gen_segemented but the library is modified slightly to compile as a thirdparty library within an Unreal Engine plugin or module.
 * **gen_c_library**: The library is heavily modifed into C11 compliant code. A segemented and single-header set of variants are generatd.
 
-
 Code not making up the core library is located in `auxiliary/<auxiliary_name>.<hpp/cpp>`. These are optional extensions or tools for the library.
 
-
 *Note: A variant of the C++ library could be generated where those additonal support features are removed (see gen_c_library implementation for an idea of how)*
 
+## Dependencies
+
+The project has no external dependencies beyond:
+
+* `errno.h`
+* `stat.h`
+* `stdarg.h`
+* `stddef.h`
+* `stdio.h`
+* `copyfile.h` (Mac)
+* `types.h`    (Linux)
+* `sys/man.h`  (Linux)
+* `fcntl.h`    (POSXIX Filesystem)
+* `unistd.h`   (Linux/Mac)
+* `intrin.h`   (Windows)
+* `io.h`       (Windows with gcc)
+* `windows.h`  (Windows)
+
+Dependencies for the project are wrapped within `GENCPP_ROLL_OWN_DEPENDENCIES` (Defining it will disable them).
+The majority of the dependency's implementation was derived from the [c-zpl library](https://github.com/zpl-c/zpl).
+
+See the following files for any updates:
+
+* [`platform.hpp`](./dependencies/platform.hpp)
+* [`src_start.cpp`](./dependencies/src_start.cpp)
+* [`filesystem.cpp`](./dependencies/filesystem.cpp)
+* [`memory.cpp`](./dependencies/memory.cpp)
+
+## Conventions
+
+This library was written in a subset of C++ where the following are not used at all:
+
+* RAII (Constructors/Destructors), lifetimes are managed using named static or regular functions.
+* Language provide dynamic dispatch, RTTI
+* Object-Oriented Inheritance
+* Exceptions
+
+Polymorphic & Member-functions are used as an ergonomic choice, along with a conserative use of operator overloads.
+The base library itself does not use anything but C-like features to allow for generating a derviative compatiable with C (WIP).
+
+## C++ template usage
+
+There are only 4 template definitions in the entire library (C++ versions). (`Array<Type>`, `Hashtable<Type>`, `swap<Type>`, and `tmpl_cast<CodeT>(CodeT code)`)
+
+Two generic templated containers are used throughout the library:
+
+* `template< class Type> struct Array`
+* `template< class Type> struct HashTable`
+
+`tmpl_cast<CodeT>(CodeT code)` is just an alternative way to explicitly cast to code. Its usage is wrapped in a macro called `cast` for the base library (needed for interoperability with C).
+
+`template< class Type> swap( Type& a, Type& b)` is used over a macro.
+
+Otherwise the library is free of any templates.
+
+## Macro usage
+
+Since this is a meta-programming library, it was desired to keep both templates and macros (especially macros) usage very limited.
+
+Most macros are defined within [macros.hpp](./dependencies/macros.hpp). 
+
+The most advanced macro usage is `num_args` which is a helper for counting the number of arguments of another macro.
+
+Any large macros used implementing the gen interface or parser are going to be phased out in favor of just forcinlined functions.  
+*(Unless there is a hot-path that requires them)*
+
+The vast majority of macros should be single-line subsitutions that either add:
+
+* Improvements to searching
+* Inteniality of keyword usage
+* A  feature that only the preprocessor has (ex: function name reflection or stringifying)
+* Compatibility of statements or expressions bewteen C & C++ that cannot be parsed by gencpp itself.
+* Masking highly verbose syntax (the latter is getting phased out).
+
+[gen_c_library](../gen_c_library/) has the most advanced set of macros for c11's generic selection.
+
+* A significant amount of explicit code geneeration is utilized to keep abuse of the preprocessor to the absolute minimum.
+* There is a heavy set of documentation inlined wth them; their naming is also highly verbose and explicit.
+* See its documentation for more information.
+
+## On base code generation
+
+There are ***five*** header files which are automatically generated by [base_codegen.hpp](./helpers/base_codegen.hpp). They are all located in [components/gen](./components/gen/).
+
+* [`ecode.hpp`](./components/gen/ecode.hpp): `CodeType` enum definition and related implementaiton. Generation is based off of [`ECodeType.csv](./enums/ECodeTypes.csv).
+* [`especifier.hpp`](./components/gen/especifier.hpp): `Specifier` enum definition, etc. Generated using [`ESpecifier.csv`](./enums/ESpecifier.csv).
+* [`eoperator.hpp`](./components/gen/eoperator.hpp): `Operator` enum definition, etc. Generated using [`EOperator.hpp`](./enums/EOperator.csv).
+* [`etoktype.cpp`](./components/gen/etoktype.cpp): `TokType` enum defininition, etc. Used by the lexer and parser backend. Uses two csvs:
+  * [`ETokType.csv`](./enums/ETokType.csv): Provides the enum entries and their strinng ids.
+  * [`AttributeTokens.csv`](./enums/AttributeTokens.csv): Provides tokens entries that should be considered as attributes  by the lexer and parser. Sspecfiically macro attributes such as those use for exporting symbols.
+* [`ast_inlines.hpp`](./components/gen/ast_inlines.hpp): Member trivial `operator` definitions for C++ code types. Does not use a csv.
+
 ## On multi-threading
 
 Currently unsupported. I want the library to be *stable* and *correct*, with the addition of exhausting all basic single-threaded optimizations before I consider multi-threading.

base/components/interface.cpp

@@ -240,22 +240,22 @@ void init()
 	}
 
 	if (Allocator_DataArrays.Proc == nullptr) {
-		Allocator_DataArrays = heap();
+		Allocator_DataArrays = GlobalAllocator;
 	}
 	if (Allocator_CodePool.Proc == nullptr ) {
-		Allocator_CodePool = heap();
+		Allocator_CodePool = GlobalAllocator;
 	}
 	if (Allocator_Lexer.Proc == nullptr) {
-		Allocator_Lexer = heap();
+		Allocator_Lexer = GlobalAllocator;
 	}
 	if (Allocator_StringArena.Proc == nullptr) {
-		Allocator_StringArena = heap();
+		Allocator_StringArena = GlobalAllocator;
 	}
 	if (Allocator_StringTable.Proc == nullptr) {
-		Allocator_StringTable = heap();
+		Allocator_StringTable = GlobalAllocator;
 	}
 	if (Allocator_TypeTable.Proc == nullptr) {
-		Allocator_TypeTable = heap();
+		Allocator_TypeTable = GlobalAllocator;
 	}
 
 	// Setup the arrays

base/components/interface.hpp

@@ -60,7 +60,7 @@ struct Opts_def_struct {
 	s32            num_interfaces;
 	ModuleFlag     mflags;
 };
-CodeClass def_class( StrC name, Opts_def_struct otps GEN_PARAM_DEFAULT );
+CodeClass def_class( StrC name, Opts_def_struct opts GEN_PARAM_DEFAULT );
 
 struct Opts_def_constructor {
 	CodeParam params;
@@ -69,7 +69,10 @@ struct Opts_def_constructor {
 };
 CodeConstructor def_constructor( Opts_def_constructor opts GEN_PARAM_DEFAULT );
 
-CodeDefine def_define( StrC name, StrC content );
+struct Opts_def_define {
+	b32 dont_append_preprocess_defines;
+};
+CodeDefine def_define( StrC name, StrC content, Opts_def_define opts GEN_PARAM_DEFAULT );
 
 struct Opts_def_destructor {
 	Code           body;

base/components/interface.parsing.cpp

@@ -8,6 +8,8 @@
 
 // Publically Exposed Interface
 
+void parser_define_macro( StrC )
+
 CodeClass parse_class( StrC def )
 {
 	GEN_USING_NS_PARSER;

base/components/interface.upfront.cpp

@@ -417,7 +417,6 @@ OpValidateResult operator__validate( Operator op, CodeParam params_code, CodeTyp
 	return InvalidCode;
 #pragma endregion Helper Marcos
 
-
 /*
 The implementaiton of the upfront constructors involves doing three things:
 * Validate the arguments given to construct the intended type of AST is valid.
@@ -616,7 +615,7 @@ CodeClass def_class( StrC name, Opts_def_struct p )
 	return result;
 }
 
-CodeDefine def_define( StrC name, StrC content )
+CodeDefine def_define( StrC name, StrC content, Opts_def_define p )
 {
 	name_check( def_define, name );
 
@@ -640,6 +639,17 @@ CodeDefine def_define( StrC name, StrC content )
 	else
 		result->Content = get_cached_string( string_to_strc(string_fmt_buf(GlobalAllocator, "%SC\n", content)) );
 
+	b32 append_preprocess_defines = ! p.dont_append_preprocess_defines;
+	if ( append_preprocess_defines ) { 
+		// Add the define to PreprocessorDefines for usage in parsing
+		s32 lex_id_len = 0;
+		for (; lex_id_len < result->Name.Len; ++ lex_id_len ) {
+			if ( reuslt->Name.Ptr[lex_id_len] == '(' )
+				break;
+		}
+		StrC lex_id = { lex_id_len, result->Name.Ptr };
+		array_append(PreprocessorDefines, lex_id );
+	}
 	return result;
 }
 

base/components/static_data.cpp

@@ -1,6 +1,6 @@
 #ifdef GEN_INTELLISENSE_DIRECTIVES
 #pragma once
-#include "gen.hpp"
+#include "../gen.hpp"
 #endif
 
 #pragma region StaticData

base/helpers/misc.hpp

@@ -47,6 +47,7 @@ void clang_format_file( char const* path, char const* style_path )
 	system( command );
 	log_fmt("\tclang-format finished formatting.\n");
 }
+
 // Will refactor a file with the given script at the provided path.
 // Assumes refactor is defined in an user-exposed or system enviornment PATH.
 // (See: ./gencpp/scripts/build.ci.ps1 for how)
@@ -65,6 +66,8 @@ void refactor_file( char const* path, char const* refactor_script )
 	#undef refactor
 }
 
+// Does either of the above or both to the provided code.
+// Code returned will be untyped content (its be serialized)
 Code code_refactor_and_format( Code code, char const* scratch_path, char const* refactor_script, char_const* clang_format_sytle_path )
 {
 	GEN_ASSERT_NOT_NULL(code);