ed/gencpp
1
0
Fork 0
mirror of https://github.com/Ed94/gencpp.git synced 2024-12-22 15:54:45 -08:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #8 from Ed94/parser_context_stack

Browse Source 
Parser context stack
This commit is contained in:
Edward R. Gonzalez 2023-07-29 17:16:04 -04:00 committed by GitHub
commit 3d7cb85e71
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
84 changed files with 1600 additions and 1267 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

566
Readme.md
View File

@ -5,11 +5,6 @@ An attempt at simple staged metaprogramming for c/c++.
The library API is a composition of code element constructors.
These build up a code AST to then serialize with a file builder.
General goal is to have a less than 15k sloc library that takes at most a couple of hours to learn and make use of.
*Why 15k ?* Assuming a seasoned coder of C++ can read and understand around 1000-2000 lines of code per hour, 15,000 could be understood in under 16-18 hours
and have confidence in modifying for their use case.
This code base attempts follow the [handmade philosophy](https://handmade.network/manifesto),
its not meant to be a black box metaprogramming utility, its meant for the user to extend for their project domain.
@ -17,43 +12,12 @@ its not meant to be a black box metaprogramming utility, its meant for the user
* [Notes](#notes)
* [Usage](#usage)
* [Building](#notes)
* [Outline](#outline)
* [What is not provided](#what-is-not-provided)
* [The three constructors ](#there-are-three-sets-of-interfaces-for-code-ast-generation-the-library-provides)
* [Predefined Codes](#predefined-codes)
* [Code generation and modification](#code-generation-and-modification)
* [Building](#building)
## Notes
The project has reached an *alpha* state, all the current functionality works for the test cases but it will most likely break in many other cases.
The project has no external dependencies beyond:
* `errno.h` (gen.dep.cpp)
* `stat.h` (gen.dep.cpp)
* `stdarg.h` (gen.dep.hpp)
* `stddef.h` (gen.dep.hpp
* `stdio.h` (gen.dep.cpp)
* `copyfile.h` (Mac, gen.dep.cpp)
* `types.h` (Linux, gen.dep.cpp)
* `unistd.h` (Linux/Mac, gen.dep.cpp)
* `intrin.h` (Windows, gen.dep.hpp)
* `io.h` (Windows with gcc, gen.dep.cpp)
* `windows.h` (Windows, gen.dep.cpp)
Dependencies for the project are wrapped within `GENCPP_ROLL_OWN_DEPENDENCIES` (Defining it will disable them).
The majority of the dependency's implementation was derived from the [c-zpl library](https://github.com/zpl-c/zpl).
This library was written a subset of C++ where the following are not used at all:
* RAII (Constructors/Destructors), lifetimes are managed using named static or regular functions.
* Language provide dynamic dispatch, RTTI
* Object-Oriented Inheritance
* Exceptions
Polymorphic & Member-functions are used as an ergonomic choice, along with a conserative use of operator overloads.
There are only 4 template definitions in the entire library. (`Array<Type>`, `Hashtable<Type>`, `swap<Type>`, and `AST/Code::cast<Type>`)
The [issues](https://github.com/Ed94/gencpp/issues) marked with v1.0 Feature indicate whats left before the library is considered feature complete.
A `natvis` and `natstepfilter` are provided in the scripts directory.
@ -92,7 +56,7 @@ u32 gen_main()
```
The design uses a constructive builder API for the code to generate.
The user is given `Code` objects that are used to build up the AST.
The user is provided `Code` objects that are used to build up the AST.
Example using each construction interface:
@ -159,528 +123,8 @@ struct ArrayHeader
```
**Note: The formatting shown here is not how it will look. For your desired formatting its recommended to run a pass through the files with an auto-formatter.**
*(The library currently uses clang-format for formatting, beaware its pretty slow...)*
## Building
An example of building is provided within project, singleheader, and test.
(All generated files go to a corresponding `*/gen` directory)
**Project**
`gen.bootstrap.cpp` generates a segmented version of the library where code is divided into `gen.hpp`, `gen.cpp`, `gen_dep.hpp`, and `gen_dep.cpp`.
(Similar whats there already, but with the includes injected)
**Singleheader**
`gen.singleheader.cpp` generates a single-header version of the library following the convention shown in popular libraries such as: gb, stb, and zpl.
**Test**
There are two meson build files the one within test is the program's build specification.
The other one in the gen directory within test is the metaprogram's build specification.
Both of them build the same source file: `test.cpp`. The only differences are that gen needs a different relative path to the include directories and defines the macro definition: `GEN_TIME`.
This method is setup where all the metaprogram's code are the within the same files as the regular program's code.
## Outline
### *WHAT IS NOT PROVIDED*
* Exceptions
* Execution statement validation : Execution expressions are defined using the untyped API.
* Lambdas (This naturally means its unsupported)
* RAII : This needs support for constructors/destructor parsing
* Haven't gotten around to yet, only useful (to me) for third-party scanning
Keywords kept from "Modern C++":
* constexpr : Great to store compile-time constants.
* consteval : Technically fine, need to make sure to execute in moderation.
* constinit : Better than constexpr at doing its job, however, its only c++ 20.
* export : Useful if c++ modules ever come around to actually being usable.
* import : ^^
* module : ^^
When it comes to expressions:
**There is no support for validating expressions.**
Its difficult to parse without enough benefits (At the metaprogramming level).
When it comes to templates:
Only trivial template support is provided. the intention is for only simple, non-recursive substitution.
The parameters of the template are treated like regular parameter AST entries.
This means that the typename entry for the parameter AST would be either:
* `class`
* `typename`
* A fundamental type, function, or pointer type.
Anything beyond this usage is not supported by parse_template for arguments (at least not intentionally).
Use at your own mental peril.
*Concepts and Constraints are not supported, its usage is non-trivial substitution.*
### The Data & Interface
As mentioned in [Usage](#usage), the user is provided Code objects by calling the constructor's functions to generate them or find existing matches.
The AST is managed by the library and provided the user via its interface.
However, the user may specifiy memory configuration.
Data layout of AST struct:
```cpp
union {
struct
{
AST* Attributes; // Class, Enum, Function, Struct, Typedef, Union, Using, Variable
AST* Specs; // Function, Operator, Type symbol, Variable
union {
AST* ParentType; // Class, Struct
AST* ReturnType; // Function, Operator
AST* UnderlyingType; // Enum, Typedef
AST* ValueType; // Parameter, Variable
};
AST* Params; // Function, Operator, Template
union {
AST* ArrExpr; // Type Symbol
AST* Body; // Class, Enum, Function, Namespace, Struct, Union
AST* Declaration; // Friend, Template
AST* Value; // Parameter, Variable
};
};
StringCached Content; // Attributes, Comment, Execution, Include
SpecifierT ArrSpecs[AST::ArrSpecs_Cap]; // Specifiers
};
union {
AST* Prev;
AST* Front; // Used by CodeBody
AST* Last; // Used by CodeParam
};
union {
AST* Next;
AST* Back; // Used by CodeBody
};
AST* Parent;
StringCached Name;
CodeT Type;
ModuleFlag ModuleFlags;
union {
OperatorT Op;
AccessSpec ParentAccess;
s32 NumEntries;
};
```
*`CodeT` is a typedef for `ECode::Type` which has an underlying type of `u32`*
*`OperatorT` is a typedef for `EOperator::Type` which has an underlying type of `u32`*
*`StringCahced` is a typedef for `String const`, to denote it is an interned string*
*`String` is the dynamically allocated string type for the library*
AST widths are setup to be AST_POD_Size.
The width dictates how much the static array can hold before it must give way to using an allocated array:
```cpp
constexpr static
uw ArrSpecs_Cap =
(
AST_POD_Size
- sizeof(AST*) * 3
- sizeof(StringCached)
- sizeof(CodeT)
- sizeof(ModuleFlag)
- sizeof(s32)
)
/ sizeof(SpecifierT) -1; // -1 for 4 extra bytes (Odd num of AST*)
```
*Ex: If the AST_POD_Size is 128 the capacity of the static array is 20.*
Data Notes:
* The allocator definitions used are exposed to the user incase they want to dictate memory usage
* You'll find the memory handling in `init`, `gen_string_allocator`, `get_cached_string`, `make_code`.
* ASTs are wrapped for the user in a Code struct which is a wrapper for a AST* type.
* Both AST and Code have member symbols but their data layout is enforced to be POD types.
* This library treats memory failures as fatal.
* Cached Strings are stored in their own set of arenas. AST constructors use cached strings for names, and content.
* `StringArenas`, `StringCache`, `Allocator_StringArena`, and `Allocator_StringTable` are the associated containers or allocators.
* Strings used for serialization and file buffers are not contained by those used for cached strings.
* They are currently using `GlobalAllocator`, which are tracked array of arenas that grows as needed (adds buckets when one runs out).
* Memory within the buckets is not reused, so its inherently wasteful.
* I will be augmenting the single arena with a simple slag allocator.
* Linked lists used children nodes on bodies, and parameters.
* Its intended to generate the AST in one go and serialize after. The constructors and serializer are designed to be a "one pass, front to back" setup.
* Allocations can be tuned by defining the folloiwng macros:
* `GEN_GLOBAL_BUCKET_SIZE` : Size of each bucket area for the global allocator
* `GEN_CODEPOOL_NUM_BLOCKS` : Number of blocks per code pool in the code allocator
* `GEN_SIZE_PER_STRING_ARENA` : Size per arena used with string caching.
* `GEN_MAX_COMMENT_LINE_LENGTH` : Longest length a comment can have per line.
* `GEN_MAX_NAME_LENGTH` : Max length of any identifier.
* `GEN_MAX_UNTYPED_STR_LENGTH` : Max content length for any untyped code.
* `GEN_TOKEN_FMT_TOKEN_MAP_MEM_SIZE` : token_fmt_va uses local_persit memory of this size for the hashtable.
* `GEN_LEX_ALLOCATOR_SIZE`
* `GEN_BUILDER_STR_BUFFER_RESERVE`
Two generic templated containers are used throughout the library:
* `template< class Type> struct Array`
* `template< class Type> struct HashTable`
Both Code and AST definitions have a `template< class Type> Code/AST cast()`. Its just an alternative way to explicitly cast to each other.
`template< class Type> swap( Type& a, Type& b)` is used over a macro.
Otherwise the library is free of any templates.
The following CodeTypes are used which the user may optionally use strong typing with if they enable: `GEN_ENFORCE_STRONG_CODE_TYPES`
* CodeBody : Has support for `for-range` iterating across Code objects.
* CodeAttributes
* CodeComment
* CodeClass
* CodeEnum
* CodeExec
* CodeExtern
* CodeInclude
* CodeFriend
* CodeFn
* CodeModule
* CodeNamespace
* CodeOperator
* CodeOpCast
* CodeParam : Has support for `for-range` iterating across parameters.
* CodeSpecifiers : Has support for `for-range` iterating across specifiers.
* CodeStruct
* CodeTemplate
* CodeType
* CodeTypedef
* CodeUnion
* CodeUsing
* CodeUsingNamespace
* CodeVar
Each Code boy has an associated "filtered AST" with the naming convention: `AST_<CodeName>`
Unrelated fields of the AST for that node type are omitted and only necessary padding members are defined otherwise.
Retrieving a raw version of the ast can be done using the `raw()` function defined in each AST.
## There are three sets of interfaces for Code AST generation the library provides
* Upfront
* Parsing
* Untyped
### Upfront Construction
All component ASTs must be previously constructed, and provided on creation of the code AST.
The construction will fail and return Code::Invalid otherwise.
Interface :``
* def_attributes
* *This is pre-appended right before the function symbol, or placed after the class or struct keyword for any flavor of attributes used.*
* *Its up to the user to use the desired attribute formatting: `[[]]` (standard), `__declspec` (Microsoft), or `__attribute__` (GNU).*
* def_comment
* def_class
* def_enum
* def_execution
* *This is equivalent to untyped_str, except that its intended for use only in execution scopes.*
* def_extern_link
* def_friend
* def_function
* def_include
* def_module
* def_namespace
* def_operator
* def_operator_cast
* def_param
* def_params
* def_specifier
* def_specifiers
* def_struct
* def_template
* def_type
* def_typedef
* def_union
* def_using
* def_using_namespace
* def_variable
Bodies:
* def_body
* def_class_body
* def_enum_body
* def_export_body
* def_extern_link_body
* def_function_body
* *Use this for operator bodies as well*
* def_global_body
* def_namespace_body
* def_struct_body
* def_union_body
Usage:
```cpp
<name> = def_<function type>( ... );
Code <name>
{
...
<name> = def_<function name>( ... );
}
```
When using the body functions, its recommended to use the args macro to auto determine the number of arguments for the varadic:
```cpp
def_global_body( args( ht_entry, array_ht_entry, hashtable ));
// instead of:
def_global_body( 3, ht_entry, array_ht_entry, hashtable );
```
If a more incremental approach is desired for the body ASTs, `Code def_body( CodeT type )` can be used to create an empty body.
When the members have been populated use: `AST::validate_body` to verify that the members are valid entires for that type.
### Parse construction
A string provided to the API is parsed for the intended language construct.
Interface :
* parse_class
* parse_enum
* parse_export_body
* parse_extern_link
* parse_friend
* Purposefully are only support forward declares with this constructor.
* parse_function
* parse_global_body
* parse_namespace
* parse_operator
* parse_operator_cast
* parse_struct
* parse_template
* parse_type
* parse_typedef
* parse_union
* parse_using
* parse_variable
The lexing and parsing takes shortcuts from whats expected in the standard.
* Numeric literals are not check for validity.
* The parse API treats any execution scope definitions with no validation and are turned into untyped Code ASTs.
* *This includes the assignment of variables.*
* Attributes ( `[[]]` (standard), `__declspec` (Microsoft), or `__attribute__` (GNU) )
* Assumed to *come before specifiers* (`const`, `constexpr`, `extern`, `static`, etc) for a function
* Or in the usual spot for class, structs, (*right after the declaration keyword*)
* typedefs have attributes with the type (`parse_type`)
* As a general rule; if its not available from the upfront constructors, its not available in the parsing constructors.
* *Upfront constructors are not necessarily used in the parsing constructors, this is just a good metric to know what can be parsed.*
* Parsing attributes can be extended to support user defined macros by defining `GEN_Define_Attribute_Tokens` (see `gen.hpp` for the formatting)
Usage:
```cpp
Code <name> = parse_<function name>( string with code );
Code <name> = def_<function name>( ..., parse_<function name>(
<string with code>
));
Code <name> = make_<function name>( ... )
{
<name>->add( parse_<function name>(
<string with code>
));
}
```
### Untyped constructions
Code ASTs are constructed using unvalidated strings.
Interface :
* token_fmt_va
* token_fmt
* untyped_str
* untyped_fmt
* untyped_token_fmt
During serialization any untyped Code AST has its string value directly injected inline of whatever context the content existed as an entry within.
Even though these are not validated from somewhat correct c/c++ syntax or components, it doesn't mean that Untyped code can be added as any component of a Code AST:
* Untyped code cannot have children, thus there cannot be recursive injection this way.
* Untyped code can only be a child of a parent of body AST, or for values of an assignment (ex: variable assignment).
These restrictions help prevent abuse of untyped code to some extent.
Usage Conventions:
```cpp
Code <name> = def_variable( <type>, <name>, untyped_<function name>(
<string with code>
));
Code <name> = untyped_str( code(
<some code without "" quotes>
));
```
Optionally, `code_str`, and `code_fmt` macros can be used so that the code macro doesn't have to be used:
```cpp
Code <name> = code_str( <some code without "" quotes > )
```
Template metaprogramming in the traditional sense becomes possible with the use of `token_fmt` and parse constructors:
```cpp
StrC value = txt_StrC("Something");
char const* template_str = txt(
Code with <key> to replace with token_values
...
);
char const* gen_code_str = token_fmt( "key", value, template_str );
Code <name> = parse_<function name>( gen_code_str );
```
## Predefined Codes
The following are provided predefined by the library as they are commonly used:
* `access_public`
* `access_protected`
* `access_private`
* `module_global_fragment`
* `module_private_fragment`
* `pragma_once`
* `spec_const`
* `spec_consteval`
* `spec_constexpr`
* `spec_constinit`
* `spec_extern_linkage` (extern)
* `spec_final`
* `spec_global` (global macro)
* `spec_inline`
* `spec_internal_linkage` (internal macro)
* `spec_local_persist` (local_persist macro)
* `spec_mutable`
* `spec_override`
* `spec_ptr`
* `spec_ref`
* `spec_register`
* `spec_rvalue`
* `spec_static_member` (static)
* `spec_thread_local`
* `spec_virtual`
* `spec_volatile`
* `spec_type_signed`
* `spec_type_unsigned`
* `spec_type_short`
* `spec_type_long`
* `t_empty`
* `t_auto`
* `t_void`
* `t_int`
* `t_bool`
* `t_char`
* `t_wchar_t`
* `t_class`
* `t_typename`
Optionally the following may be defined if `GEN_DEFINE_LIBRARY_CODE_CONSTANTS` is defined
* `t_b32`
* `t_s8`
* `t_s16`
* `t_s32`
* `t_s64`
* `t_u8`
* `t_u16`
* `t_u32`
* `t_u64`
* `t_sw`
* `t_uw`
* `t_f32`
* `t_f64`
## Extent of operator overload validation
The AST and constructors will be able to validate that the arguments provided for the operator type match the expected form:
* If return type must match a parameter
* If number of parameters is correct
* If added as a member symbol to a class or struct, that operator matches the requirements for the class (types match up)
The user is responsible for making sure the code types provided are correct
and have the desired specifiers assigned to them beforehand.
## Code generation and modification
There are three provided file interfaces:
* Builder
* Editor
* Scanner
Editor and Scanner are disabled by default, use `GEN_FEATURE_EDITOR` and `GEN_FEATURE_SCANNER` to enable them.
### Builder is a similar object to the jai language's string_builder
* The purpose of it is to generate a file.
* A file is specified and opened for writing using the open( file_path) ) function.
* The code is provided via print( code ) function will be serialized to its buffer.
* When all serialization is finished, use the write() command to write the buffer to the file.
### Editor is for editing a series of files based on a set of requests provided to it
**Note: Not implemented yet**
* The purpose is to overrite a specific file, it places its contents in a buffer to scan.
* Requests are populated using the following interface:
* add : Add code.
* remove : Remove code.
* replace: Replace code.
All three have the same parameters with exception to remove which only has SymbolInfo and Policy:
* SymbolInfo:
* File : The file the symbol resides in. Leave null to indicate to search all files. Leave null to indicated all-file search.
* Marker : #define symbol that indicates a location or following signature is valid to manipulate. Leave null to indicate the signature should only be used.
* Signature : Use a Code symbol to find a valid location to manipulate, can be further filtered with the marker. Leave null to indicate the marker should only be used.
* Policy : Additional policy info for completing the request (empty for now)
* Code : Code to inject if adding, or replace existing code with.
Additionally if `GEN_FEATURE_EDITOR_REFACTOR` is defined, refactor( file_path, specification_path ) wil be made available.
Refactor is based of the refactor library and uses its interface.
It will on call add a request to the queue to run the refactor script on the file.
### Scanner allows the user to generate Code ASTs by reading files
**Note: Not implemented yet**
* The purpose is to grab definitions to generate metadata or generate new code from these definitions.
* Requests are populated using the add( SymbolInfo, Policy ) function. The symbol info is the same as the one used for the editor. So is the case with Policy.
The file will only be read from, no writing supported.
One great use case is for example: generating the single-header library for gencpp!
### Additional Info (Editor and Scanner)
When all requests have been populated, call process_requests().
It will provide an output of receipt data of the results when it completes.
Files may be added to the Editor and Scanner additionally with add_files( num, files ).
This is intended for when you have requests that are for multiple files.
Request queue in both Editor and Scanner are cleared once process_requests completes.
See the [scripts directory](scripts/).

49
docs/Parsing.md Normal file
View File

@ -0,0 +1,49 @@
# Parsing
The library features a naive parser tailored for only what the library needs to construct the supported syntax of C++ into its AST.
This parser does not, and should not do the compiler's job. By only supporting this minimal set of features, the parser is kept under 5000 loc.
Everything is done in one pass for both the preprocessor directives and the rest of the language.
The parser performs no macro expansion as the scope of gencpp feature-set is to only support the preprocessor for the goal of having rudimentary awareness of preprocessor ***conditionals***, ***defines***, and ***includes***.
*(Conditionals and defines are a TODO)*
The keywords supported for the preprocessor are:
* include
* define
* if
* ifdef
* elif
* endif
* undef
Just like with actual preprocessor, each directive # line is considered one preproecessor unit, and will be treated as one Preprocessor AST. *These ASTs will be considered members or entries of braced scope they reside within*.
All keywords except *include* are suppported as members of a scope for a class/struct, global, or namespace body.
Any preprocessor definition abuse that changes the syntax of the core language is unsupported and will fail to parse if not kept within an execution scope (function body, or expression assignment).
The parsing implementation supports the following for the user:
```cpp
CodeClass parse_class ( StrC class_def );
CodeEnum parse_enum ( StrC enum_def );
CodeBody parse_export_body ( StrC export_def );
CodeExtern parse_extern_link ( StrC exten_link_def);
CodeFriend parse_friend ( StrC friend_def );
CodeFn parse_function ( StrC fn_def );
CodeBody parse_global_body ( StrC body_def );
CodeNamespace parse_namespace ( StrC namespace_def );
CodeOperator parse_operator ( StrC operator_def );
CodeOpCast parse_operator_cast( StrC operator_def );
CodeStruct parse_struct ( StrC struct_def );
CodeTemplate parse_template ( StrC template_def );
CodeType parse_type ( StrC type_def );
CodeTypedef parse_typedef ( StrC typedef_def );
CodeUnion parse_union ( StrC union_def );
CodeUsing parse_using ( StrC using_def );
CodeVar parse_variable ( StrC var_def );
```
***Parsing will aggregate any tokens within a function body or expression statement to an untyped Code AST.***

537
docs/Readme.md Normal file
View File

@ -0,0 +1,537 @@
## Documentation
The project has no external dependencies beyond:
* `errno.h`
* `stat.h`
* `stdarg.h`
* `stddef.h`
* `stdio.h`
* `copyfile.h` (Mac)
* `types.h` (Linux)
* `unistd.h` (Linux/Mac)
* `intrin.h` (Windows)
* `io.h` (Windows with gcc)
* `windows.h` (Windows)
Dependencies for the project are wrapped within `GENCPP_ROLL_OWN_DEPENDENCIES` (Defining it will disable them).
The majority of the dependency's implementation was derived from the [c-zpl library](https://github.com/zpl-c/zpl).
This library was written in a subset of C++ where the following are not used at all:
* RAII (Constructors/Destructors), lifetimes are managed using named static or regular functions.
* Language provide dynamic dispatch, RTTI
* Object-Oriented Inheritance
* Exceptions
Polymorphic & Member-functions are used as an ergonomic choice, along with a conserative use of operator overloads.
There are only 4 template definitions in the entire library. (`Array<Type>`, `Hashtable<Type>`, `swap<Type>`, and `AST/Code::cast<Type>`)
Two generic templated containers are used throughout the library:
* `template< class Type> struct Array`
* `template< class Type> struct HashTable`
Both Code and AST definitions have a `template< class Type> Code/AST cast()`. Its just an alternative way to explicitly cast to each other.
`template< class Type> swap( Type& a, Type& b)` is used over a macro.
Otherwise the library is free of any templates.
### *WHAT IS NOT PROVIDED*
* Execution statement validation : Execution expressions are defined using the untyped AST.
* Lambdas (This naturally means its unsupported)
* RAII : This needs support for constructors/destructor parsing
* Haven't gotten around to yet (its in the github issues)
Keywords kept from "Modern C++":
* constexpr : Great to store compile-time constants.
* consteval : Technically fine, need to make sure to execute in moderation.
* constinit : Better than constexpr at doing its job, however, its only c++ 20.
* export : Useful if c++ modules ever come around to actually being usable.
* import : ^^
* module : ^^
When it comes to expressions:
**There is no support for validating expressions.**
Its difficult to parse without enough benefits (At the metaprogramming level).
When it comes to templates:
**Only trivial template support is provided.**
The intention is for only simple, non-recursive substitution.
The parameters of the template are treated like regular parameter AST entries.
This means that the typename entry for the parameter AST would be either:
* `class`
* `typename`
* A fundamental type, function, or pointer type.
Anything beyond this usage is not supported by parse_template for arguments (at least not intentionally).
Use at your own mental peril.
*Concepts and Constraints are not supported, its usage is non-trivial substitution.*
### The Data & Interface
As mentioned in [Usage](#usage), the user is provided Code objects by calling the constructor's functions to generate them or find existing matches.
The AST is managed by the library and provided the user via its interface.
However, the user may specifiy memory configuration.
Data layout of AST struct:
```cpp
union {
struct
{
AST* Attributes; // Class, Enum, Function, Struct, Typedef, Union, Using, Variable
AST* Specs; // Function, Operator, Type symbol, Variable
union {
AST* ParentType; // Class, Struct
AST* ReturnType; // Function, Operator
AST* UnderlyingType; // Enum, Typedef
AST* ValueType; // Parameter, Variable
};
AST* Params; // Function, Operator, Template
union {
AST* ArrExpr; // Type Symbol
AST* Body; // Class, Enum, Function, Namespace, Struct, Union
AST* Declaration; // Friend, Template
AST* Value; // Parameter, Variable
};
};
StringCached Content; // Attributes, Comment, Execution, Include
SpecifierT ArrSpecs[AST::ArrSpecs_Cap]; // Specifiers
};
union {
AST* Prev;
AST* Front; // Used by CodeBody
AST* Last; // Used by CodeParam
};
union {
AST* Next;
AST* Back; // Used by CodeBody
};
AST* Parent;
StringCached Name;
CodeT Type;
ModuleFlag ModuleFlags;
union {
OperatorT Op;
AccessSpec ParentAccess;
s32 NumEntries;
};
```
*`CodeT` is a typedef for `ECode::Type` which has an underlying type of `u32`*
*`OperatorT` is a typedef for `EOperator::Type` which has an underlying type of `u32`*
*`StringCahced` is a typedef for `String const`, to denote it is an interned string*
*`String` is the dynamically allocated string type for the library*
AST widths are setup to be AST_POD_Size.
The width dictates how much the static array can hold before it must give way to using an allocated array:
```cpp
constexpr static
uw ArrSpecs_Cap =
(
AST_POD_Size
- sizeof(AST*) * 3
- sizeof(StringCached)
- sizeof(CodeT)
- sizeof(ModuleFlag)
- sizeof(s32)
)
/ sizeof(SpecifierT) -1; // -1 for 4 extra bytes (Odd num of AST*)
```
*Ex: If the AST_POD_Size is 128 the capacity of the static array is 20.*
Data Notes:
* The allocator definitions used are exposed to the user incase they want to dictate memory usage
* You'll find the memory handling in `init`, `gen_string_allocator`, `get_cached_string`, `make_code`.
* ASTs are wrapped for the user in a Code struct which is a wrapper for a AST* type.
* Both AST and Code have member symbols but their data layout is enforced to be POD types.
* This library treats memory failures as fatal.
* Cached Strings are stored in their own set of arenas. AST constructors use cached strings for names, and content.
* `StringArenas`, `StringCache`, `Allocator_StringArena`, and `Allocator_StringTable` are the associated containers or allocators.
* Strings used for serialization and file buffers are not contained by those used for cached strings.
* They are currently using `GlobalAllocator`, which are tracked array of arenas that grows as needed (adds buckets when one runs out).
* Memory within the buckets is not reused, so its inherently wasteful.
* I will be augmenting the single arena with a simple slag allocator.
* Linked lists used children nodes on bodies, and parameters.
* Its intended to generate the AST in one go and serialize after. The constructors and serializer are designed to be a "one pass, front to back" setup.
* Allocations can be tuned by defining the folloiwng macros:
* `GEN_BUILDER_STR_BUFFER_RESERVE`
* `GEN_CODEPOOL_NUM_BLOCKS` : Number of blocks per code pool in the code allocator
* `GEN_GLOBAL_BUCKET_SIZE` : Size of each bucket area for the global allocator
* `GEN_LEX_ALLOCATOR_SIZE`
* `GEN_MAX_COMMENT_LINE_LENGTH` : Longest length a comment can have per line.
* `GEN_MAX_NAME_LENGTH` : Max length of any identifier.
* `GEN_MAX_UNTYPED_STR_LENGTH` : Max content length for any untyped code.
* `GEN_SIZE_PER_STRING_ARENA` : Size per arena used with string caching.
* `GEN_TOKEN_FMT_TOKEN_MAP_MEM_SIZE` : token_fmt_va uses local_persit memory of this size for the hashtable.
The following CodeTypes are used which the user may optionally use strong typing with if they enable: `GEN_ENFORCE_STRONG_CODE_TYPES`
* CodeBody : Has support for `for-range` iterating across Code objects.
* CodeAttributes
* CodeComment
* CodeClass
* CodeConstructor
* CodeDefine
* CodeDestructor
* CodePreprocessCond
* CodeEnum
* CodeExec
* CodeExtern
* CodeInclude
* CodeFriend
* CodeFn
* CodeModule
* CodeNamespace
* CodeOperator
* CodeOpCast
* CodeParam : Has support for `for-range` iterating across parameters.
* CodeSpecifiers : Has support for `for-range` iterating across specifiers.
* CodeStruct
* CodeTemplate
* CodeType
* CodeTypedef
* CodeUnion
* CodeUsing
* CodeVar
Each Code boy has an associated "filtered AST" with the naming convention: `AST_<CodeName>`
Unrelated fields of the AST for that node type are omitted and only necessary padding members are defined otherwise.
Retrieving a raw version of the ast can be done using the `raw()` function defined in each AST.
## There are three sets of interfaces for Code AST generation the library provides
* Upfront
* Parsing
* Untyped
### Upfront Construction
All component ASTs must be previously constructed, and provided on creation of the code AST.
The construction will fail and return Code::Invalid otherwise.
Interface :``
* def_attributes
* *This is pre-appended right before the function symbol, or placed after the class or struct keyword for any flavor of attributes used.*
* *Its up to the user to use the desired attribute formatting: `[[]]` (standard), `__declspec` (Microsoft), or `__attribute__` (GNU).*
* def_comment
* def_class
* def_constructor
* def_destructor
* def_enum
* def_execution
* *This is equivalent to untyped_str, except that its intended for use only in execution scopes.*
* def_extern_link
* def_friend
* def_function
* def_include
* def_module
* def_namespace
* def_operator
* def_operator_cast
* def_param
* def_params
* def_specifier
* def_specifiers
* def_struct
* def_template
* def_type
* def_typedef
* def_union
* def_using
* def_using_namespace
* def_variable
Bodies:
* def_body
* def_class_body
* def_enum_body
* def_export_body
* def_extern_link_body
* def_function_body
* *Use this for operator bodies as well*
* def_global_body
* def_namespace_body
* def_struct_body
* def_union_body
Usage:
```cpp
<name> = def_<function type>( ... );
Code <name>
{
...
<name> = def_<function name>( ... );
}
```
When using the body functions, its recommended to use the args macro to auto determine the number of arguments for the varadic:
```cpp
def_global_body( args( ht_entry, array_ht_entry, hashtable ));
// instead of:
def_global_body( 3, ht_entry, array_ht_entry, hashtable );
```
If a more incremental approach is desired for the body ASTs, `Code def_body( CodeT type )` can be used to create an empty body.
When the members have been populated use: `AST::validate_body` to verify that the members are valid entires for that type.
### Parse construction
A string provided to the API is parsed for the intended language construct.
Interface :
* parse_class
* parse_constructor
* parse_destructor
* parse_enum
* parse_export_body
* parse_extern_link
* parse_friend
* Purposefully are only support forward declares with this constructor.
* parse_function
* parse_global_body
* parse_namespace
* parse_operator
* parse_operator_cast
* parse_struct
* parse_template
* parse_type
* parse_typedef
* parse_union
* parse_using
* parse_variable
The lexing and parsing takes shortcuts from whats expected in the standard.
* Numeric literals are not check for validity.
* The parse API treats any execution scope definitions with no validation and are turned into untyped Code ASTs.
* *This includes the assignment of variables.*
* Attributes ( `[[]]` (standard), `__declspec` (Microsoft), or `__attribute__` (GNU) )
* Assumed to *come before specifiers* (`const`, `constexpr`, `extern`, `static`, etc) for a function
* Or in the usual spot for class, structs, (*right after the declaration keyword*)
* typedefs have attributes with the type (`parse_type`)
* As a general rule; if its not available from the upfront constructors, its not available in the parsing constructors.
* *Upfront constructors are not necessarily used in the parsing constructors, this is just a good metric to know what can be parsed.*
* Parsing attributes can be extended to support user defined macros by defining `GEN_DEFINE_ATTRIBUTE_TOKENS` (see `gen.hpp` for the formatting)
Usage:
```cpp
Code <name> = parse_<function name>( string with code );
Code <name> = def_<function name>( ..., parse_<function name>(
<string with code>
));
Code <name> = make_<function name>( ... )
{
<name>->add( parse_<function name>(
<string with code>
));
}
```
### Untyped constructions
Code ASTs are constructed using unvalidated strings.
Interface :
* token_fmt_va
* token_fmt
* untyped_str
* untyped_fmt
* untyped_token_fmt
During serialization any untyped Code AST has its string value directly injected inline of whatever context the content existed as an entry within.
Even though these are not validated from somewhat correct c/c++ syntax or components, it doesn't mean that Untyped code can be added as any component of a Code AST:
* Untyped code cannot have children, thus there cannot be recursive injection this way.
* Untyped code can only be a child of a parent of body AST, or for values of an assignment (ex: variable assignment).
These restrictions help prevent abuse of untyped code to some extent.
Usage Conventions:
```cpp
Code <name> = def_variable( <type>, <name>, untyped_<function name>(
<string with code>
));
Code <name> = untyped_str( code(
<some code without "" quotes>
));
```
Optionally, `code_str`, and `code_fmt` macros can be used so that the code macro doesn't have to be used:
```cpp
Code <name> = code_str( <some code without "" quotes > )
```
Template metaprogramming in the traditional sense becomes possible with the use of `token_fmt` and parse constructors:
```cpp
StrC value = txt_StrC("Something");
char const* template_str = txt(
Code with <key> to replace with token_values
...
);
char const* gen_code_str = token_fmt( "key", value, template_str );
Code <name> = parse_<function name>( gen_code_str );
```
## Predefined Codes
The following are provided predefined by the library as they are commonly used:
* `access_public`
* `access_protected`
* `access_private`
* `module_global_fragment`
* `module_private_fragment`
* `param_varaidc` (Used for varadic definitions)
* `preprocess_else`
* `preprocess_endif`
* `pragma_once`
* `spec_const`
* `spec_consteval`
* `spec_constexpr`
* `spec_constinit`
* `spec_extern_linkage` (extern)
* `spec_final`
* `spec_global` (global macro)
* `spec_inline`
* `spec_internal_linkage` (internal macro)
* `spec_local_persist` (local_persist macro)
* `spec_mutable`
* `spec_override`
* `spec_ptr`
* `spec_ref`
* `spec_register`
* `spec_rvalue`
* `spec_static_member` (static)
* `spec_thread_local`
* `spec_virtual`
* `spec_volatile`
* `spec_type_signed`
* `spec_type_unsigned`
* `spec_type_short`
* `spec_type_long`
* `t_empty` (Used for varaidc macros)
* `t_auto`
* `t_void`
* `t_int`
* `t_bool`
* `t_char`
* `t_wchar_t`
* `t_class`
* `t_typename`
Optionally the following may be defined if `GEN_DEFINE_LIBRARY_CODE_CONSTANTS` is defined
* `t_b32`
* `t_s8`
* `t_s16`
* `t_s32`
* `t_s64`
* `t_u8`
* `t_u16`
* `t_u32`
* `t_u64`
* `t_sw`
* `t_uw`
* `t_f32`
* `t_f64`
## Extent of operator overload validation
The AST and constructors will be able to validate that the arguments provided for the operator type match the expected form:
* If return type must match a parameter
* If number of parameters is correct
* If added as a member symbol to a class or struct, that operator matches the requirements for the class (types match up)
The user is responsible for making sure the code types provided are correct
and have the desired specifiers assigned to them beforehand.
## Code generation and modification
There are three provided file interfaces:
* Builder
* Editor
* Scanner
Editor and Scanner are disabled by default, use `GEN_FEATURE_EDITOR` and `GEN_FEATURE_SCANNER` to enable them.
### Builder is a similar object to the jai language's string_builder
* The purpose of it is to generate a file.
* A file is specified and opened for writing using the open( file_path) ) function.
* The code is provided via print( code ) function will be serialized to its buffer.
* When all serialization is finished, use the write() command to write the buffer to the file.
### Editor is for editing a series of files based on a set of requests provided to it
**Note: Not implemented yet**
* The purpose is to overrite a specific file, it places its contents in a buffer to scan.
* Requests are populated using the following interface:
* add : Add code.
* remove : Remove code.
* replace: Replace code.
All three have the same parameters with exception to remove which only has SymbolInfo and Policy:
* SymbolInfo:
* File : The file the symbol resides in. Leave null to indicate to search all files. Leave null to indicated all-file search.
* Marker : #define symbol that indicates a location or following signature is valid to manipulate. Leave null to indicate the signature should only be used.
* Signature : Use a Code symbol to find a valid location to manipulate, can be further filtered with the marker. Leave null to indicate the marker should only be used.
* Policy : Additional policy info for completing the request (empty for now)
* Code : Code to inject if adding, or replace existing code with.
Additionally if `GEN_FEATURE_EDITOR_REFACTOR` is defined, refactor( file_path, specification_path ) wil be made available.
Refactor is based of the refactor library and uses its interface.
It will on call add a request to the queue to run the refactor script on the file.
### Scanner allows the user to generate Code ASTs by reading files
**Note: Not implemented yet**
* The purpose is to grab definitions to generate metadata or generate new code from these definitions.
* Requests are populated using the add( SymbolInfo, Policy ) function. The symbol info is the same as the one used for the editor. So is the case with Policy.
The file will only be read from, no writing supported.
One great use case is for example: generating the single-header library for gencpp!
### Additional Info (Editor and Scanner)
When all requests have been populated, call process_requests().
It will provide an output of receipt data of the results when it completes.
Files may be added to the Editor and Scanner additionally with add_files( num, files ).
This is intended for when you have requests that are for multiple files.
Request queue in both Editor and Scanner are cleared once process_requests completes.

55
gencpp.vcxproj
View File

@ -112,12 +112,18 @@
</ItemGroup>
<ItemGroup>
<ClInclude Include="project\components\gen.data_structures.hpp" />
<ClInclude Include="project\components\gen.ecode.hpp" />
<ClInclude Include="project\components\gen.eoperator.hpp" />
<ClInclude Include="project\components\gen.especifier.hpp" />
<ClInclude Include="project\components\gen.header_end.hpp" />
<ClInclude Include="project\components\gen.header_start.hpp" />
<ClInclude Include="project\components\gen.interface.hpp" />
<ClInclude Include="project\components\gen.types.hpp" />
<ClInclude Include="project\dependencies\gen.basic_types.hpp" />
<ClInclude Include="project\dependencies\gen.containers.hpp" />
<ClInclude Include="project\dependencies\gen.dep.hpp" />
<ClInclude Include="project\dependencies\gen.debug.hpp" />
<ClInclude Include="project\dependencies\gen.file_handling.hpp" />
<ClInclude Include="project\dependencies\gen.hashing.hpp" />
<ClInclude Include="project\dependencies\gen.header_start.hpp" />
<ClInclude Include="project\dependencies\gen.macros.hpp" />
<ClInclude Include="project\dependencies\gen.memory.hpp" />
@ -125,25 +131,17 @@
<ClInclude Include="project\dependencies\gen.printing.hpp" />
<ClInclude Include="project\dependencies\gen.string.hpp" />
<ClInclude Include="project\dependencies\gen.string_ops.hpp" />
<ClInclude Include="project\dependencies\gen.timing.hpp" />
<ClInclude Include="project\filesystem\gen.builder.hpp" />
<ClInclude Include="project\filesystem\gen.editor.hpp" />
<ClInclude Include="project\filesystem\gen.scanner.hpp" />
<ClInclude Include="project\gen.dep.hpp" />
<ClInclude Include="project\gen.editor.hpp" />
<ClInclude Include="project\gen.hpp" />
<ClInclude Include="project\gen.pop_ignores.inline.hpp" />
<ClInclude Include="project\gen.push_ignores.inline.hpp" />
<ClInclude Include="project\gen.scanner.hpp" />
<ClInclude Include="project\gen.undef.macros.hpp" />
<ClInclude Include="project\helpers\gen.pop_ignores.inline.hpp" />
<ClInclude Include="project\helpers\gen.push_ignores.inline.hpp" />
<ClInclude Include="project\helpers\gen.undef.macros.hpp" />
<ClInclude Include="singleheader\components\gen.header_start.hpp" />
<ClInclude Include="test\DummyInclude.hpp" />
<ClInclude Include="test\gen\array.Upfront.gen.hpp" />
<ClInclude Include="test\gen\buffer.Upfront.gen.hpp" />
<ClInclude Include="test\gen\hashtable.Upfront.gen.hpp" />
<ClInclude Include="test\gen\ring.Upfront.gen.hpp" />
<ClInclude Include="test\gen\sanity.Upfront.gen.hpp" />
<ClInclude Include="test\Parsed\Buffer.Parsed.hpp" />
<ClInclude Include="test\Parsed\HashTable.Parsed.hpp" />
<ClInclude Include="test\Parsed\Ring.Parsed.hpp" />
@ -160,18 +158,28 @@
</ItemGroup>
<ItemGroup>
<ClCompile Include="project\components\gen.ast.cpp" />
<ClCompile Include="project\components\gen.ast_case_macros.cpp" />
<ClCompile Include="project\components\gen.data.cpp" />
<ClCompile Include="project\components\gen.etoktype.cpp" />
<ClCompile Include="project\components\gen.impl_start.cpp" />
<ClCompile Include="project\components\gen.interface.cpp" />
<ClCompile Include="project\components\gen.interface.parsing.cpp" />
<ClCompile Include="project\components\gen.interface.upfront.cpp" />
<ClCompile Include="project\dependencies\gen.dep.cpp" />
<ClCompile Include="project\components\gen.untyped.cpp" />
<ClCompile Include="project\dependencies\gen.debug.cpp" />
<ClCompile Include="project\dependencies\gen.file_handling.cpp" />
<ClCompile Include="project\dependencies\gen.hashing.cpp" />
<ClCompile Include="project\dependencies\gen.impl_start.cpp" />
<ClCompile Include="project\dependencies\gen.memory.cpp" />
<ClCompile Include="project\dependencies\gen.parsing.cpp" />
<ClCompile Include="project\dependencies\gen.printing.cpp" />
<ClCompile Include="project\dependencies\gen.string.cpp" />
<ClCompile Include="project\dependencies\gen.string_ops.cpp" />
<ClCompile Include="project\dependencies\gen.timing.cpp" />
<ClCompile Include="project\gen.bootstrap.cpp" />
<ClCompile Include="project\gen.cpp" />
<ClCompile Include="project\gen.dep.cpp" />
<ClCompile Include="singleheader\gen\gen.singleheader.cpp" />
<ClCompile Include="test\gen\build\meson-private\sanitycheckc.c" />
<ClCompile Include="test\gen\build\meson-private\sanitycheckcpp.cc" />
<ClCompile Include="singleheader\gen.singleheader.cpp" />
<ClCompile Include="test\parsing.cpp" />
<ClCompile Include="test\sanity.cpp" />
<ClCompile Include="test\SOA.cpp" />
@ -184,6 +192,23 @@
<Natvis Include=".vscode\gencpp.natvis" />
<Natvis Include="scripts\gencpp.natvis" />
</ItemGroup>
<ItemGroup>
<Content Include="project\components\AttributeTokens.csv" />
<Content Include="project\components\ECode.csv" />
<Content Include="project\components\EOperator.csv" />
<Content Include="project\components\ESpecifier.csv" />
<Content Include="project\components\ETokType.csv" />
<Content Include="project\meson.build" />
<Content Include="scripts\.clang-format" />
<Content Include="scripts\bootstrap.ci.ps1" />
<Content Include="scripts\bootstrap.ps1" />
<Content Include="scripts\msvc\build_msvc.ps1" />
<Content Include="scripts\msvc\devshell.ps1" />
<Content Include="scripts\refactor.ps1" />
<Content Include="scripts\singleheader.ci.ps1" />
<Content Include="scripts\singleheader.ps1" />
<Content Include="singleheader\meson.build" />
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets">
</ImportGroup>

40
project/Readme.md
View File

@ -1,47 +1,43 @@
# Documentation
The core library is contained within `gen.hpp` and `gen.cpp`.
Things related to the editor and scanner are in their own respective files. (Ex: `gen.scanner.<hpp/cpp>` )
Dependencies are within `gen.dep.<hpp/cpp>`
The library is fragmented into a series of headers and sources files meant to be scanned in and then generated to a tailored format for the target
`gen` files.
The principal (user) files are `gen.hpp` and `gen.cpp`.
They contain includes for its various components: `components/<component_name>.<hpp/cpp>`
Dependencies are bundled into `gen.dep.<hpp/cpp>`.
Just like the `gen.<hpp/cpp>` they include their components: `dependencies/<dependency_name>.<hpp/cpp>`
The fle processors are in their own respective files. (Ex: `file_processors/<file_processor>.<hpp/cpp>` )
They directly include `depedencies/file_handling.<hpp/cpp>` as the core library does not include file processing by defualt.
**TODO : Right now the library is not finished structurally, as such the first self-hosting iteration is still WIP**
Both libraries use *pre-generated* (self-hosting I guess) version of the library to then generate the latest version of itself.
(sort of a verification that the generated version is equivalent)
(sort of a verification that the generated version is equivalent).
The default `gen.bootstrap.cpp` located in the project folder is meant to be produce a standard segmented library, where the components of the library
have relatively dedicated header and source files. With dependencies included at the top of the file and each header starting with a pragma once.
This will overwrite the existing library implementation in the immediate directory.
have relatively dedicated header and source files. Dependencies included at the top of the file and each header starting with a pragma once.
The output will be in the `project/gen` directory (if the directory does not exist, it will create it).
Use those to get a general idea of how to make your own tailored version.
If the naming convention is undesired, the `gencpp.refactor` script can be used with the [refactor]()
Feature Macros:
* `GEN_DEFINE_ATTRIBUTE_TOKENS` : Allows user to define their own attribute macros for use in parsing.
* This is auto-generated if using the bootstrap or single-header generation
* *Note: The user will use the `AttributeTokens.csv` when the library is fully self-hosting.*
* `GEN_DEFINE_LIBRARY_CORE_CONSTANTS` : Optional typename codes as they are non-standard to C/C++ and not necessary to library usage
* `GEN_DONT_USE_NAMESPACE` : By default, the library is wrapped in a `gen` namespace, this will disable that expose it to the global scope.
* `GEN_DONT_ENFORCE_GEN_TIME_GUARD` : By default, the library ( gen.hpp/ gen.cpp ) expects the macro `GEN_TIME` to be defined, this disables that.
* `GEN_ROLL_OWN_DEPENDENCIES` : Optional override so that user may define the dependencies themselves.
* `GEN_DEFINE_LIBRARY_CORE_CONSTANTS` : Optional typename codes as they are non-standard to C/C++ and not necessary to library usage
* `GEN_ENFORCE_STRONG_CODE_TYPES` : Enforces casts to filtered code types.
* `GEN_EXPOSE_BACKEND` : Will expose symbols meant for internal use only.
* `GEN_Define_Attribute_Tokens` : Allows user to define their own attribute macros for use in parsing.
`GEN_USE_RECURSIVE_AST_DUPLICATION` is available but its not well tested and should not need to be used.
If constructing ASTs properly. There should be no modification of ASTs, and thus this would never become an issue.
(I will probably remove down the line...)
* `GEN_ROLL_OWN_DEPENDENCIES` : Optional override so that user may define the dependencies themselves.
## On multi-threading
Currently unsupported. The following changes would have to be made:
* Setup static data access with fences if more than one thread will generate ASTs ( or keep a different set for each thread)
* Make sure local persistent data of functions are also thread local.
* The builder should be done on a per-thread basis.
* Due to the design of the editor and scanner, it will most likely be best to make each file a job to process request entries on. Receipts should have an an array to store per thread. They can be combined to the final receipts array when all files have been processed.
## Extending the library
This library is relatively very small, and can be extended without much hassle.

85
project/components/gen.ast.cpp → project/components/ast.cpp
View File

@ -17,20 +17,9 @@ AST* AST::duplicate()
String AST::to_string()
{
# define ProcessModuleFlags() \
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export )) \
result.append( "export " ); \
\
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Import )) \
result.append( "import " ); \
local_persist thread_local
char SerializationLevel = 0;
#if defined(GEN_BENCHMARK) && defined(GEN_BENCHMARK_SERIALIZATION)
u64 time_start = time_rel_ms();
#endif
// TODO : Need to refactor so that intermeidate strings are freed conviently.
String result = String::make( GlobalAllocator, "" );
@ -82,7 +71,8 @@ String AST::to_string()
case Class:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
if ( Attributes || ParentType )
{
@ -129,7 +119,8 @@ String AST::to_string()
case Class_Fwd:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
if ( Attributes )
result.append_fmt( "class %s %s;", Attributes->to_string(), Name );
@ -140,7 +131,8 @@ String AST::to_string()
case Enum:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
if ( Attributes || UnderlyingType )
{
@ -170,7 +162,8 @@ String AST::to_string()
case Enum_Fwd:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
if ( Attributes )
result.append_fmt( "%s ", Attributes->to_string() );
@ -181,7 +174,8 @@ String AST::to_string()
case Enum_Class:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
if ( Attributes || UnderlyingType )
{
@ -219,7 +213,8 @@ String AST::to_string()
case Enum_Class_Fwd:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
result.append( "enum class " );
@ -262,7 +257,8 @@ String AST::to_string()
case Function:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
if ( Attributes )
result.append_fmt( "%s ", Attributes->to_string() );
@ -301,7 +297,8 @@ String AST::to_string()
case Function_Fwd:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
if ( Attributes )
result.append_fmt( "%s ", Attributes->to_string() );
@ -347,7 +344,8 @@ String AST::to_string()
break;
case Namespace:
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
result.append_fmt( "namespace %s\n{\n%s}"
, Name
@ -358,7 +356,8 @@ String AST::to_string()
case Operator:
case Operator_Member:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
if ( Attributes )
result.append_fmt( "%s ", Attributes->to_string() );
@ -395,7 +394,8 @@ String AST::to_string()
case Operator_Fwd:
case Operator_Member_Fwd:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
if ( Attributes )
result.append_fmt( "%s ", Attributes->to_string() );
@ -513,7 +513,8 @@ String AST::to_string()
case Struct:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
if ( Name == nullptr)
{
@ -566,7 +567,8 @@ String AST::to_string()
case Struct_Fwd:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
if ( Attributes )
result.append_fmt( "struct %s %s;", Attributes->to_string(), Name );
@ -577,7 +579,8 @@ String AST::to_string()
case Template:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
result.append_fmt( "template< %s >\n%s", Params->to_string(), Declaration->to_string() );
}
@ -585,7 +588,8 @@ String AST::to_string()
case Typedef:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
result.append( "typedef ");
@ -624,7 +628,8 @@ String AST::to_string()
case Union:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
result.append( "union " );
@ -650,7 +655,8 @@ String AST::to_string()
case Using:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
if ( Attributes )
result.append_fmt( "%s ", Attributes->to_string() );
@ -675,7 +681,8 @@ String AST::to_string()
case Variable:
{
ProcessModuleFlags();
if ( bitfield_is_equal( u32, ModuleFlags, ModuleFlag::Export ))
result.append( "export " );
if ( Attributes || Specs )
{
@ -726,11 +733,7 @@ String AST::to_string()
break;
}
#if defined(GEN_BENCHMARK) && defined(GEN_BENCHMARK_SERIALIZATION)
log_fmt("AST::to_string() time taken: %llu for: %s\n", time_rel_ms() - time_start, result );
#endif
return result;
#undef ProcessModuleFlags
}
bool AST::is_equal( AST* other )
@ -778,7 +781,7 @@ bool AST::validate_body()
switch ( Type )
{
case Class_Body:
CheckEntries( AST_BODY_CLASS_UNALLOWED_TYPES );
CheckEntries( GEN_AST_BODY_CLASS_UNALLOWED_TYPES );
break;
case Enum_Body:
for ( Code entry : cast<CodeBody>() )
@ -791,22 +794,22 @@ bool AST::validate_body()
}
break;
case Export_Body:
CheckEntries( AST_BODY_CLASS_UNALLOWED_TYPES );
CheckEntries( GEN_AST_BODY_CLASS_UNALLOWED_TYPES );
break;
case Extern_Linkage:
CheckEntries( AST_BODY_EXTERN_LINKAGE_UNALLOWED_TYPES );
CheckEntries( GEN_AST_BODY_EXTERN_LINKAGE_UNALLOWED_TYPES );
break;
case Function_Body:
CheckEntries( AST_BODY_FUNCTION_UNALLOWED_TYPES );
CheckEntries( GEN_AST_BODY_FUNCTION_UNALLOWED_TYPES );
break;
case Global_Body:
CheckEntries( AST_BODY_GLOBAL_UNALLOWED_TYPES );
CheckEntries( GEN_AST_BODY_GLOBAL_UNALLOWED_TYPES );
break;
case Namespace_Body:
CheckEntries( AST_BODY_NAMESPACE_UNALLOWED_TYPES );
CheckEntries( GEN_AST_BODY_NAMESPACE_UNALLOWED_TYPES );
break;
case Struct_Body:
CheckEntries( AST_BODY_STRUCT_UNALLOWED_TYPES );
CheckEntries( GEN_AST_BODY_STRUCT_UNALLOWED_TYPES );
break;
case Union_Body:
for ( Code entry : Body->cast<CodeBody>() )
@ -825,6 +828,8 @@ bool AST::validate_body()
}
return false;
#undef CheckEntries
}
#pragma endregion AST

79
project/components/ast_case_macros.cpp Normal file
View File

@ -0,0 +1,79 @@
# define GEN_AST_BODY_CLASS_UNALLOWED_TYPES \
case PlatformAttributes: \
case Class_Body: \
case Enum_Body: \
case Extern_Linkage: \
case Function_Body: \
case Function_Fwd: \
case Global_Body: \
case Namespace: \
case Namespace_Body: \
case Operator: \
case Operator_Fwd: \
case Parameters: \
case Specifiers: \
case Struct_Body: \
case Typename:
# define GEN_AST_BODY_STRUCT_UNALLOWED_TYPES GEN_AST_BODY_CLASS_UNALLOWED_TYPES
# define GEN_AST_BODY_FUNCTION_UNALLOWED_TYPES \
case Access_Public: \
case Access_Protected: \
case Access_Private: \
case PlatformAttributes: \
case Class_Body: \
case Enum_Body: \
case Extern_Linkage: \
case Friend: \
case Function_Body: \
case Function_Fwd: \
case Global_Body: \
case Namespace: \
case Namespace_Body: \
case Operator: \
case Operator_Fwd: \
case Operator_Member: \
case Operator_Member_Fwd: \
case Parameters: \
case Specifiers: \
case Struct_Body: \
case Typename:
# define GEN_AST_BODY_GLOBAL_UNALLOWED_TYPES \
case Access_Public: \
case Access_Protected: \
case Access_Private: \
case PlatformAttributes: \
case Class_Body: \
case Enum_Body: \
case Execution: \
case Friend: \
case Function_Body: \
case Global_Body: \
case Namespace_Body: \
case Operator_Member: \
case Operator_Member_Fwd: \
case Parameters: \
case Specifiers: \
case Struct_Body: \
case Typename:
# define GEN_AST_BODY_EXPORT_UNALLOWED_TYPES GEN_AST_BODY_GLOBAL_UNALLOWED_TYPES
# define GEN_AST_BODY_EXTERN_LINKAGE_UNALLOWED_TYPES GEN_AST_BODY_GLOBAL_UNALLOWED_TYPES
# define GEN_AST_BODY_NAMESPACE_UNALLOWED_TYPES \
case Access_Public: \
case Access_Protected: \
case Access_Private: \
case PlatformAttributes: \
case Class_Body: \
case Enum_Body: \
case Execution: \
case Friend: \
case Function_Body: \
case Namespace_Body: \
case Operator_Member: \
case Operator_Member_Fwd: \
case Parameters: \
case Specifiers: \
case Struct_Body: \
case Typename:

0
project/components/gen.data_structures.hpp → project/components/data_structures.hpp
View File

0
project/components/gen.ecode.hpp → project/components/ecode.hpp
View File

0
project/components/gen.eoperator.hpp → project/components/eoperator.hpp
View File

1
project/components/gen.especifier.hpp → project/components/especifier.hpp
View File

@ -101,5 +101,4 @@ namespace ESpecifier
# undef Define_Specifiers
}
using SpecifierT = ESpecifier::Type;

28
project/components/gen.etoktype.cpp → project/components/etoktype.cpp
View File

@ -9,6 +9,12 @@ namespace Parser
Attributes_Start is only used to indicate the start of the user_defined attribute list.
*/
#ifndef GEN_DEFINE_ATTRIBUTE_TOKENS
# define GEN_DEFINE_ATTRIBUTE_TOKENS \
Entry( API_Export, "GEN_API_Export_Code" ) \
Entry( API_Import, "GEN_API_Import_Code" )
#endif
# define Define_TokType \
Entry( Invalid, "INVALID" ) \
Entry( Access_Private, "private" ) \
@ -27,8 +33,8 @@ namespace Parser
Entry( BraceSquare_Close, "]" ) \
Entry( Capture_Start, "(" ) \
Entry( Capture_End, ")" ) \
Entry( Comment, "__comment__" ) \
Entry( Char, "__char__" ) \
Entry( Comment, "comment" ) \
Entry( Char, "character" ) \
Entry( Comma, "," ) \
Entry( Decl_Class, "class" ) \
Entry( Decl_GNU_Attribute, "__attribute__" ) \
@ -44,13 +50,17 @@ namespace Parser
Entry( Decl_Typedef, "typedef" ) \
Entry( Decl_Using, "using" ) \
Entry( Decl_Union, "union" ) \
Entry( Identifier, "__identifier__" ) \
Entry( Identifier, "identifier" ) \
Entry( Module_Import, "import" ) \
Entry( Module_Export, "export" ) \
Entry( Number, "number" ) \
Entry( Operator, "operator" ) \
Entry( Preprocessor_Directive, "#") \
Entry( Preprocessor_Include, "include" ) \
Entry( Preprocess_Define, "#define") \
Entry( Preproces_Include, "include" ) \
Entry( Preprocess_If, "#if") \
Entry( Preprocess_Elif, "#elif") \
Entry( Preprocess_Else, "#else") \
Entry( Preprocess_EndIf, "#endif") \
Entry( Spec_Alignas, "alignas" ) \
Entry( Spec_Const, "const" ) \
Entry( Spec_Consteval, "consteval" ) \
@ -70,7 +80,7 @@ namespace Parser
Entry( Spec_Volatile, "volatile") \
Entry( Star, "*" ) \
Entry( Statement_End, ";" ) \
Entry( String, "__string__" ) \
Entry( String, "string" ) \
Entry( Type_Unsigned, "unsigned" ) \
Entry( Type_Signed, "signed" ) \
Entry( Type_Short, "short" ) \
@ -87,7 +97,7 @@ namespace Parser
{
# define Entry( Name_, Str_ ) Name_,
Define_TokType
GEN_Define_Attribute_Tokens
GEN_DEFINE_ATTRIBUTE_TOKENS
# undef Entry
NumTokens,
};
@ -100,7 +110,7 @@ namespace Parser
{
# define Entry( Name_, Str_ ) { sizeof(Str_), Str_ },
Define_TokType
GEN_Define_Attribute_Tokens
GEN_DEFINE_ATTRIBUTE_TOKENS
# undef Entry
};
@ -127,7 +137,7 @@ namespace Parser
{
# define Entry( Name_, Str_ ) Str_,
Define_TokType
GEN_Define_Attribute_Tokens
GEN_DEFINE_ATTRIBUTE_TOKENS
# undef Entry
};

81
project/components/gen.ast_case_macros.cpp
View File

@ -1,81 +0,0 @@
# define AST_BODY_CLASS_UNALLOWED_TYPES \
case PlatformAttributes: \
case Class_Body: \
case Enum_Body: \
case Extern_Linkage: \
case Function_Body: \
case Function_Fwd: \
case Global_Body: \
case Namespace: \
case Namespace_Body: \
case Operator: \
case Operator_Fwd: \
case Parameters: \
case Specifiers: \
case Struct_Body: \
case Typename:
# define AST_BODY_FUNCTION_UNALLOWED_TYPES \
case Access_Public: \
case Access_Protected: \
case Access_Private: \
case PlatformAttributes: \
case Class_Body: \
case Enum_Body: \
case Extern_Linkage: \
case Friend: \
case Function_Body: \
case Function_Fwd: \
case Global_Body: \
case Namespace: \
case Namespace_Body: \
case Operator: \
case Operator_Fwd: \
case Operator_Member: \
case Operator_Member_Fwd: \
case Parameters: \
case Specifiers: \
case Struct_Body: \
case Typename:
# define AST_BODY_GLOBAL_UNALLOWED_TYPES \
case Access_Public: \
case Access_Protected: \
case Access_Private: \
case PlatformAttributes: \
case Class_Body: \
case Enum_Body: \
case Execution: \
case Friend: \
case Function_Body: \
case Global_Body: \
case Namespace_Body: \
case Operator_Member: \
case Operator_Member_Fwd: \
case Parameters: \
case Specifiers: \
case Struct_Body: \
case Typename:
# define AST_BODY_EXPORT_UNALLOWED_TYPES AST_BODY_GLOBAL_UNALLOWED_TYPES
# define AST_BODY_NAMESPACE_UNALLOWED_TYPES \
case Access_Public: \
case Access_Protected: \
case Access_Private: \
case PlatformAttributes: \
case Class_Body: \
case Enum_Body: \
case Execution: \
case Friend: \
case Function_Body: \
case Namespace_Body: \
case Operator_Member: \
case Operator_Member_Fwd: \
case Parameters: \
case Specifiers: \
case Struct_Body: \
case Typename:
# define AST_BODY_EXTERN_LINKAGE_UNALLOWED_TYPES AST_BODY_GLOBAL_UNALLOWED_TYPES
# define AST_BODY_STRUCT_UNALLOWED_TYPES AST_BODY_CLASS_UNALLOWED_TYPES

9
project/components/gen.header_end.hpp → project/components/header_end.hpp
View File

@ -84,6 +84,7 @@ Code& Code::operator ++()
}
#pragma region AST & Code Gen Common
#define Define_CodeImpl( Typename ) \
char const* Typename::debug_str() \
{ \
@ -243,6 +244,7 @@ Define_CodeCast( Using );
Define_CodeCast( Var );
Define_CodeCast( Body);
#undef Define_CodeCast
#pragma endregion AST & Code Gen Common
void CodeClass::add_interface( CodeType type )
@ -361,9 +363,11 @@ StrC token_fmt_impl( sw num, ... )
return { result, buf };
}
#pragma endregion Inlines
#pragma region Constants
#ifdef GEN_DEFINE_LIBRARY_CODE_CONSTANTS
// Predefined typename codes. Are set to readonly and are setup during gen::init()
@ -477,9 +481,11 @@ extern CodeSpecifiers spec_static_member;
extern CodeSpecifiers spec_thread_local;
extern CodeSpecifiers spec_virtual;
extern CodeSpecifiers spec_volatile;
#pragma endregion Constants
#pragma region Macros
# define gen_main main
# define __ NoCode
@ -498,9 +504,11 @@ extern CodeSpecifiers spec_volatile;
// Takes a format string (char const*) and a list of tokens (StrC) and returns a StrC of the formatted string.
# define token_fmt( ... ) gen::token_fmt_impl( (num_args( __VA_ARGS__ ) + 1) / 2, __VA_ARGS__ )
#pragma endregion Macros
#ifdef GEN_EXPOSE_BACKEND
// Global allocator used for data with process lifetime.
extern AllocatorInfo GlobalAllocator;
extern Array< Arena > Global_AllocatorBuckets;
@ -517,4 +525,5 @@ extern CodeSpecifiers spec_volatile;
extern AllocatorInfo Allocator_StringArena;
extern AllocatorInfo Allocator_StringTable;
extern AllocatorInfo Allocator_TypeTable;
#endif

0
project/components/gen.header_start.hpp → project/components/header_start.hpp
View File

0
project/components/gen.impl_start.cpp → project/components/impl_start.cpp
View File

0
project/components/gen.interface.cpp → project/components/interface.cpp
View File

0
project/components/gen.interface.hpp → project/components/interface.hpp
View File

832
project/components/gen.interface.parsing.cpp → project/components/interface.parsing.cpp
View File

File diff suppressed because it is too large Load Diff

28
project/components/gen.interface.upfront.cpp → project/components/interface.upfront.cpp
View File

@ -1338,7 +1338,7 @@ CodeBody def_class_body( s32 num, ... )
va_list va;
va_start(va, num);
def_body_code_validation_start( def_class_body );
AST_BODY_CLASS_UNALLOWED_TYPES
GEN_AST_BODY_CLASS_UNALLOWED_TYPES
def_body_code_validation_end( def_class_body );
va_end(va);
@ -1354,7 +1354,7 @@ CodeBody def_class_body( s32 num, Code* codes )
result->Type = Function_Body;
def_body_code_array_validation_start( def_class_body );
AST_BODY_CLASS_UNALLOWED_TYPES
GEN_AST_BODY_CLASS_UNALLOWED_TYPES
def_body_code_validation_end( def_class_body );
return result;
@ -1437,7 +1437,7 @@ CodeBody def_export_body( s32 num, ... )
va_list va;
va_start(va, num);
def_body_code_validation_start( def_export_body );
AST_BODY_EXPORT_UNALLOWED_TYPES
GEN_AST_BODY_EXPORT_UNALLOWED_TYPES
def_body_code_validation_end( def_export_body );
va_end(va);
@ -1453,7 +1453,7 @@ CodeBody def_export_body( s32 num, Code* codes )
result->Type = Export_Body;
def_body_code_array_validation_start( def_export_body );
AST_BODY_EXPORT_UNALLOWED_TYPES
GEN_AST_BODY_EXPORT_UNALLOWED_TYPES
def_body_code_validation_end( def_export_body );
return result;
@ -1470,7 +1470,7 @@ CodeBody def_extern_link_body( s32 num, ... )
va_list va;
va_start(va, num);
def_body_code_validation_start( def_extern_linkage_body );
AST_BODY_EXTERN_LINKAGE_UNALLOWED_TYPES
GEN_AST_BODY_EXTERN_LINKAGE_UNALLOWED_TYPES
def_body_code_validation_end( def_extern_linkage_body );
va_end(va);
@ -1486,7 +1486,7 @@ CodeBody def_extern_link_body( s32 num, Code* codes )
result->Type = Extern_Linkage_Body;
def_body_code_array_validation_start( def_extern_linkage_body );
AST_BODY_EXTERN_LINKAGE_UNALLOWED_TYPES
GEN_AST_BODY_EXTERN_LINKAGE_UNALLOWED_TYPES
def_body_code_validation_end( def_extern_linkage_body );
return result;
@ -1503,7 +1503,7 @@ CodeBody def_function_body( s32 num, ... )
va_list va;
va_start(va, num);
def_body_code_validation_start( def_function_body );
AST_BODY_FUNCTION_UNALLOWED_TYPES
GEN_AST_BODY_FUNCTION_UNALLOWED_TYPES
def_body_code_validation_end( def_function_body );
va_end(va);
@ -1519,7 +1519,7 @@ CodeBody def_function_body( s32 num, Code* codes )
result->Type = Function_Body;
def_body_code_array_validation_start( def_function_body );
AST_BODY_FUNCTION_UNALLOWED_TYPES
GEN_AST_BODY_FUNCTION_UNALLOWED_TYPES
def_body_code_validation_end( def_function_body );
return result;
@ -1536,7 +1536,7 @@ CodeBody def_global_body( s32 num, ... )
va_list va;
va_start(va, num);
def_body_code_validation_start( def_global_body );
AST_BODY_GLOBAL_UNALLOWED_TYPES
GEN_AST_BODY_GLOBAL_UNALLOWED_TYPES
def_body_code_validation_end( def_global_body );
va_end(va);
@ -1552,7 +1552,7 @@ CodeBody def_global_body( s32 num, Code* codes )
result->Type = Global_Body;
def_body_code_array_validation_start( def_global_body );
AST_BODY_GLOBAL_UNALLOWED_TYPES
GEN_AST_BODY_GLOBAL_UNALLOWED_TYPES
def_body_code_validation_end( def_global_body );
return result;
@ -1569,7 +1569,7 @@ CodeBody def_namespace_body( s32 num, ... )
va_list va;
va_start(va, num);
def_body_code_validation_start( def_namespace_body );
AST_BODY_NAMESPACE_UNALLOWED_TYPES
GEN_AST_BODY_NAMESPACE_UNALLOWED_TYPES
def_body_code_validation_end( def_namespace_body );
va_end(va);
@ -1585,7 +1585,7 @@ CodeBody def_namespace_body( s32 num, Code* codes )
result->Type = Global_Body;
def_body_code_array_validation_start( def_namespace_body );
AST_BODY_NAMESPACE_UNALLOWED_TYPES
GEN_AST_BODY_NAMESPACE_UNALLOWED_TYPES
def_body_code_validation_end( def_namespace_body );
return result;
@ -1737,7 +1737,7 @@ CodeBody def_struct_body( s32 num, ... )
va_list va;
va_start(va, num);
def_body_code_validation_start( def_struct_body );
AST_BODY_STRUCT_UNALLOWED_TYPES
GEN_AST_BODY_STRUCT_UNALLOWED_TYPES
def_body_code_validation_end( def_struct_body );
va_end(va);
@ -1753,7 +1753,7 @@ CodeBody def_struct_body( s32 num, Code* codes )
result->Type = Struct_Body;
def_body_code_array_validation_start( def_struct_body );
AST_BODY_STRUCT_UNALLOWED_TYPES
GEN_AST_BODY_STRUCT_UNALLOWED_TYPES
def_body_code_validation_end( def_struct_body );
return result;

4
project/components/gen.data.cpp → project/components/static_data.cpp
View File

@ -1,4 +1,5 @@
#pragma region StaticData
// TODO : Convert global allocation strategy to use a slab allocation strategy.
global AllocatorInfo GlobalAllocator;
global Array<Arena> Global_AllocatorBuckets;
@ -16,9 +17,11 @@ global AllocatorInfo Allocator_Lexer = heap();
global AllocatorInfo Allocator_StringArena = heap();
global AllocatorInfo Allocator_StringTable = heap();
global AllocatorInfo Allocator_TypeTable = heap();
#pragma endregion StaticData
#pragma region Constants
global CodeType t_empty;
global CodeType t_auto;
global CodeType t_void;
@ -83,4 +86,5 @@ global CodeSpecifiers spec_static_member;
global CodeSpecifiers spec_thread_local;
global CodeSpecifiers spec_virtual;
global CodeSpecifiers spec_volatile;
#pragma endregion Constants

9
project/components/gen.types.hpp → project/components/types.hpp
View File

@ -68,7 +68,7 @@ ModuleFlag operator|( ModuleFlag A, ModuleFlag B)
Override these to change the attribute to your own unique identifier convention.
The tokenizer identifies attribute defines with the GEN_Define_Attribute_Tokens macros.
The tokenizer identifies attribute defines with the GEN_DEFINE_ATTRIBUTE_TOKENS macros.
See the example below and the Define_TokType macro used in gen.cpp to know the format.
While the library can parse raw attributes, most projects use defines to wrap them for compiler
platform indendence. The token define allows support for them without having to modify the library.
@ -100,10 +100,3 @@ constexpr char const* Attribute_Keyword = stringize( GEN_Attribute_Keyword );
constexpr char const* Attribute_Keyword = "";
#endif
#ifndef GEN_Define_Attribute_Tokens
# define GEN_Define_Attribute_Tokens \
Entry( API_Export, "GEN_API_Export_Code" ) \
Entry( API_Import, "GEN_API_Import_Code" )
#endif

1
project/components/gen.untyped.cpp → project/components/untyped.cpp
View File

@ -12,7 +12,6 @@ sw token_fmt_va( char* buf, uw buf_size, s32 num_tokens, va_list va )
char tok_map_mem[ TokenFmt_TokenMap_MemSize ];
tok_map_arena = Arena::init_from_memory( tok_map_mem, sizeof(tok_map_mem) );
tok_map = HashTable<StrC>::init( tok_map_arena );
s32 left = num_tokens - 1;

0
project/dependencies/gen.basic_types.hpp → project/dependencies/basic_types.hpp
View File

0
project/dependencies/gen.containers.hpp → project/dependencies/containers.hpp
View File

2
project/dependencies/gen.debug.cpp → project/dependencies/debug.cpp
View File

@ -5,7 +5,7 @@ void assert_handler( char const* condition, char const* file, s32 line, char con
_printf_err( "%s:(%d): Assert Failure: ", file, line );
if ( condition )
_printf_err( "`%s` ", condition );
_printf_err( "`%s` \n", condition );
if ( msg )
{

0
project/dependencies/gen.debug.hpp → project/dependencies/debug.hpp
View File

0
project/dependencies/gen.file_handling.cpp → project/dependencies/file_handling.cpp
View File

0
project/dependencies/gen.file_handling.hpp → project/dependencies/file_handling.hpp
View File

0
project/dependencies/gen.hashing.cpp → project/dependencies/hashing.cpp
View File

0
project/dependencies/gen.hashing.hpp → project/dependencies/hashing.hpp
View File

0
project/dependencies/gen.header_start.hpp → project/dependencies/header_start.hpp
View File

0
project/dependencies/gen.impl_start.cpp → project/dependencies/impl_start.cpp
View File

0
project/dependencies/gen.macros.hpp → project/dependencies/macros.hpp
View File

0
project/dependencies/gen.memory.cpp → project/dependencies/memory.cpp
View File

0
project/dependencies/gen.memory.hpp → project/dependencies/memory.hpp
View File

0
project/dependencies/gen.parsing.cpp → project/dependencies/parsing.cpp
View File

0
project/dependencies/gen.parsing.hpp → project/dependencies/parsing.hpp
View File

0
project/dependencies/gen.printing.cpp → project/dependencies/printing.cpp
View File

7
project/dependencies/gen.printing.hpp → project/dependencies/printing.hpp
View File

@ -10,12 +10,15 @@ struct FileInfo;
char* str_fmt_buf ( char const* fmt, ... );
char* str_fmt_buf_va ( char const* fmt, va_list va );
sw str_fmt_va ( char* str, sw n, char const* fmt, va_list va );
sw str_fmt_file ( FileInfo* f, char const* fmt, ... );
sw str_fmt_file_va ( FileInfo* f, char const* fmt, va_list va );
sw str_fmt_out_va ( char const* fmt, va_list va );
sw str_fmt_out_err ( char const* fmt, ... );
sw str_fmt_out_err_va( char const* fmt, va_list va );
// TODO : Move these to file handling.
sw str_fmt_file ( FileInfo* f, char const* fmt, ... );
sw str_fmt_file_va ( FileInfo* f, char const* fmt, va_list va );
constexpr
char const* Msg_Invalid_Value = "INVALID VALUE PROVIDED";

0
project/dependencies/gen.string.cpp → project/dependencies/string.cpp
View File

0
project/dependencies/gen.string.hpp → project/dependencies/string.hpp
View File

0
project/dependencies/gen.string_ops.cpp → project/dependencies/string_ops.cpp
View File

0
project/dependencies/gen.string_ops.hpp → project/dependencies/string_ops.hpp
View File

0
project/dependencies/gen.timing.cpp → project/dependencies/timing.cpp
View File

0
project/dependencies/gen.timing.hpp → project/dependencies/timing.hpp
View File

0
project/components/AttributeTokens.csv → project/enums/AttributeTokens.csv
View File

0
project/components/ECode.csv → project/enums/ECode.csv
View File

0
project/components/EOperator.csv → project/enums/EOperator.csv
View File

0
project/components/ESpecifier.csv → project/enums/ESpecifier.csv
View File

16
project/components/ETokType.csv → project/enums/ETokType.csv
View File

@ -15,8 +15,8 @@ BraceSquare_Open, "["
BraceSquare_Close, "]"
Capture_Start, "("
Capture_End, ")"
Comment, "__comment__"
Char, "__char__"
Comment, "comemnt"
Char, "character"
Comma, ","
Decl_Class, "class"
Decl_GNU_Attribute, "__attribute__"
@ -32,13 +32,17 @@ Decl_Template, "template"
Decl_Typedef, "typedef"
Decl_Using, "using"
Decl_Union, "union"
Identifier, "__identifier__"
Identifier, "identifier"
Module_Import, "import"
Module_Export, "export"
Number, "number"
Operator, "operator"
Preprocessor_Directive, "#"
Preprocessor_Include, "include"
Preprocess_Define, "#define"
Preprocess_Include, "#include"
Preprocess_If, "#if"
Preprocess_ElIF, "#elif"
Preprocess_Else, "#else"
Preprocess_EndIf, "#endif"
Spec_Alignas, "alignas"
Spec_Const, "const"
Spec_Consteval, "consteval"
@ -58,7 +62,7 @@ Spec_ThreadLocal, "thread_local"
Spec_Volatile, "volatile"
Star, "*"
Statement_End, ";"
String, "__string__"
String, "string"
Type_Unsigned, "unsigned"
Type_Signed, "signed"
Type_Short, "short"
1 Invalid __invalid__
15 BraceSquare_Close ]
16 Capture_Start (
17 Capture_End )
18 Comment __comment__ comemnt
19 Char __char__ character
20 Comma ,
21 Decl_Class class
22 Decl_GNU_Attribute __attribute__
32 Decl_Typedef typedef
33 Decl_Using using
34 Decl_Union union
35 Identifier __identifier__ identifier
36 Module_Import import
37 Module_Export export
38 Number number
39 Operator operator
40 Preprocessor_Directive Preprocess_Define # #define
41 Preprocessor_Include Preprocess_Include include #include
42 Preprocess_If #if
43 Preprocess_ElIF #elif
44 Preprocess_Else #else
45 Preprocess_EndIf #endif
46 Spec_Alignas alignas
47 Spec_Const const
48 Spec_Consteval consteval
62 Spec_Volatile volatile
63 Star *
64 Statement_End ;
65 String __string__ string
66 Type_Unsigned unsigned
67 Type_Signed signed
68 Type_Short short

0
project/filesystem/gen.builder.cpp → project/file_processors/builder.cpp
View File

0
project/filesystem/gen.builder.hpp → project/file_processors/builder.hpp
View File

0
project/filesystem/gen.editor.hpp → project/file_processors/editor.hpp
View File

0
project/filesystem/gen.scanner.hpp → project/file_processors/scanner.hpp
View File

94
project/gen.bootstrap.cpp
View File

@ -2,8 +2,8 @@
#define GEN_ENFORCE_STRONG_CODE_TYPES
#define GEN_EXPOSE_BACKEND
#include "gen.cpp"
#include "filesystem/gen.scanner.hpp"
#include "helpers/gen.helper.hpp"
#include "file_processors/scanner.hpp"
#include "helpers/helper.hpp"
using namespace gen;
@ -33,25 +33,27 @@ int gen_main()
{
gen::init();
Code push_ignores = scan_file( "helpers/gen.push_ignores.inline.hpp" );
Code pop_ignores = scan_file( "helpers/gen.pop_ignores.inline.hpp" );
Code push_ignores = scan_file( "helpers/push_ignores.inline.hpp" );
Code pop_ignores = scan_file( "helpers/pop_ignores.inline.hpp" );
// gen_dep.hpp
{
Code header_start = scan_file( "dependencies/gen.header_start.hpp" );
Code header_start = scan_file( "dependencies/header_start.hpp" );
Code nspace_macro = untyped_str( namespace_by_default ? nspace_default : nspace_non_default );
Code macros = scan_file( "dependencies/gen.macros.hpp" );
Code basic_types = scan_file( "dependencies/gen.basic_types.hpp" );
Code debug = scan_file( "dependencies/gen.debug.hpp" );
Code memory = scan_file( "dependencies/gen.memory.hpp" );
Code string_ops = scan_file( "dependencies/gen.string_ops.hpp" );
Code printing = scan_file( "dependencies/gen.printing.hpp" );
Code containers = scan_file( "dependencies/gen.containers.hpp" );
Code hashing = scan_file( "dependencies/gen.hashing.hpp" );
Code string = scan_file( "dependencies/gen.string.hpp" );
Code file_handling = scan_file( "dependencies/gen.file_handling.hpp" );
Code parsing = scan_file( "dependencies/gen.parsing.hpp" );
Code timing = scan_file( "dependencies/gen.timing.hpp" );
Code macros = scan_file( "dependencies/macros.hpp" );
Code basic_types = scan_file( "dependencies/basic_types.hpp" );
Code debug = scan_file( "dependencies/debug.hpp" );
Code memory = scan_file( "dependencies/memory.hpp" );
Code string_ops = scan_file( "dependencies/string_ops.hpp" );
Code printing = scan_file( "dependencies/printing.hpp" );
Code containers = scan_file( "dependencies/containers.hpp" );
Code hashing = scan_file( "dependencies/hashing.hpp" );
Code string = scan_file( "dependencies/string.hpp" );
Code parsing = scan_file( "dependencies/parsing.hpp" );
Code timing = scan_file( "dependencies/timing.hpp" );
// TOOD : Make this optional
Code file_handling = scan_file( "dependencies/file_handling.hpp" );
Builder
deps_header;
@ -82,15 +84,15 @@ int gen_main()
// gen_dep.cpp
{
CodeInclude header = def_include( txt_StrC("gen_dep.hpp") );
Code impl_start = scan_file( "dependencies/gen.impl_start.cpp" );
Code debug = scan_file( "dependencies/gen.debug.cpp" );
Code string_ops = scan_file( "dependencies/gen.string_ops.cpp" );
Code printing = scan_file( "dependencies/gen.printing.cpp" );
Code memory = scan_file( "dependencies/gen.memory.cpp" );
Code parsing = scan_file( "dependencies/gen.parsing.cpp" );
Code hashing = scan_file( "dependencies/gen.hashing.cpp" );
Code string = scan_file( "dependencies/gen.string.cpp" );
Code timing = scan_file( "dependencies/gen.timing.cpp" );
Code impl_start = scan_file( "dependencies/impl_start.cpp" );
Code debug = scan_file( "dependencies/debug.cpp" );
Code string_ops = scan_file( "dependencies/string_ops.cpp" );
Code printing = scan_file( "dependencies/printing.cpp" );
Code memory = scan_file( "dependencies/memory.cpp" );
Code parsing = scan_file( "dependencies/parsing.cpp" );
Code hashing = scan_file( "dependencies/hashing.cpp" );
Code string = scan_file( "dependencies/string.cpp" );
Code timing = scan_file( "dependencies/timing.cpp" );
Builder
deps_impl;
@ -115,18 +117,19 @@ int gen_main()
// gen.hpp
{
Code header_start = scan_file( "components/gen.header_start.hpp" );
Code header_start = scan_file( "components/header_start.hpp" );
Code nspace_macro = untyped_str( namespace_by_default ? nspace_default : nspace_non_default );
Code types = scan_file( "components/gen.types.hpp" );
Code data_structs = scan_file( "components/gen.data_structures.hpp" );
Code interface = scan_file( "components/gen.interface.hpp" );
Code header_end = scan_file( "components/gen.header_end.hpp" );
Code types = scan_file( "components/types.hpp" );
Code data_structs = scan_file( "components/data_structures.hpp" );
Code interface = scan_file( "components/interface.hpp" );
Code header_end = scan_file( "components/header_end.hpp" );
CodeBody ecode = gen_ecode( "./components/ECode.csv" );
CodeBody eoperator = gen_eoperator( "./components/EOperator.csv" );
CodeBody especifier = gen_especifier( "./components/ESpecifier.csv" );
CodeBody ecode = gen_ecode( "enums/ECode.csv" );
CodeBody eoperator = gen_eoperator( "enums/EOperator.csv" );
CodeBody especifier = gen_especifier( "enums/ESpecifier.csv" );
Code builder = scan_file( "filesystem/gen.builder.hpp" );
// TODO : Make this optional to include
Code builder = scan_file( "file_processors/builder.hpp" );
Builder
header;
@ -157,20 +160,21 @@ int gen_main()
// gen.cpp
{
Code impl_start = scan_file( "components/gen.impl_start.cpp" );
Code impl_start = scan_file( "components/impl_start.cpp" );
CodeInclude header = def_include( txt_StrC("gen.hpp") );
Code data = scan_file( "components/gen.data.cpp" );
Code ast_case_macros = scan_file( "components/gen.ast_case_macros.cpp" );
Code ast = scan_file( "components/gen.ast.cpp" );
Code interface = scan_file( "components/gen.interface.cpp" );
Code upfront = scan_file( "components/gen.interface.upfront.cpp" );
Code parsing = scan_file( "components/gen.interface.parsing.cpp" );
Code untyped = scan_file( "components/gen.untyped.cpp" );
Code data = scan_file( "components/static_data.cpp" );
Code ast_case_macros = scan_file( "components/ast_case_macros.cpp" );
Code ast = scan_file( "components/ast.cpp" );
Code interface = scan_file( "components/interface.cpp" );
Code upfront = scan_file( "components/interface.upfront.cpp" );
Code parsing = scan_file( "components/interface.parsing.cpp" );
Code untyped = scan_file( "components/untyped.cpp" );
CodeBody etoktype = gen_etoktype( "components/ETokType.csv", "components/AttributeTokens.csv" );
CodeBody etoktype = gen_etoktype( "enums/ETokType.csv", "enums/AttributeTokens.csv" );
CodeNamespace parser_nspace = def_namespace( name(Parser), def_namespace_body( args(etoktype)) );
Code builder = scan_file( "filesystem/gen.builder.cpp" );
// TODO : Make this optional to include
Code builder = scan_file( "file_processors/builder.cpp" );
Builder
impl;

22
project/gen.cpp
View File

@ -1,4 +1,4 @@
#include "helpers/gen.push_ignores.inline.hpp"
#include "helpers/push_ignores.inline.hpp"
// ReSharper disable CppClangTidyClangDiagnosticSwitchEnum
@ -16,19 +16,19 @@
GEN_NS_BEGIN
#include "components/gen.data.cpp"
#include "components/static_data.cpp"
#include "components/gen.ast_case_macros.cpp"
#include "components/gen.ast.cpp"
#include "components/ast_case_macros.cpp"
#include "components/ast.cpp"
#include "components/gen.interface.cpp"
#include "components/gen.interface.upfront.cpp"
#include "components/gen.etoktype.cpp"
#include "components/gen.interface.parsing.cpp"
#include "components/gen.untyped.cpp"
#include "components/interface.cpp"
#include "components/interface.upfront.cpp"
#include "components/etoktype.cpp"
#include "components/interface.parsing.cpp"
#include "components/untyped.cpp"
#include "filesystem/gen.builder.cpp"
#include "file_processors/builder.cpp"
GEN_NS_END
#include "helpers/gen.pop_ignores.inline.hpp"
#include "helpers/pop_ignores.inline.hpp"

20
project/gen.dep.cpp
View File

@ -1,19 +1,19 @@
// This file is intended to be included within gen.cpp (There is no pragma diagnostic ignores)
#include "gen.dep.hpp"
#include "dependencies/gen.impl_start.cpp"
#include "dependencies/impl_start.cpp"
GEN_NS_BEGIN
#include "dependencies/gen.debug.cpp"
#include "dependencies/gen.string_ops.cpp"
#include "dependencies/gen.printing.cpp"
#include "dependencies/gen.memory.cpp"
#include "dependencies/gen.parsing.cpp"
#include "dependencies/gen.hashing.cpp"
#include "dependencies/gen.string.cpp"
#include "dependencies/gen.timing.cpp"
#include "dependencies/debug.cpp"
#include "dependencies/string_ops.cpp"
#include "dependencies/printing.cpp"
#include "dependencies/memory.cpp"
#include "dependencies/parsing.cpp"
#include "dependencies/hashing.cpp"
#include "dependencies/string.cpp"
#include "dependencies/timing.cpp"
#include "dependencies/gen.file_handling.cpp"
#include "dependencies/file_handling.cpp"
GEN_NS_END

26
project/gen.dep.hpp
View File

@ -1,7 +1,7 @@
// This file is intended to be included within gen.hpp (There is no pragma diagnostic ignores)
#pragma once
#include "dependencies/gen.header_start.hpp"
#include "dependencies/header_start.hpp"
#ifdef GEN_DONT_USE_NAMESPACE
# define GEN_NS_BEGIN
@ -13,16 +13,18 @@
GEN_NS_BEGIN
#include "dependencies/gen.macros.hpp"
#include "dependencies/gen.basic_types.hpp"
#include "dependencies/gen.debug.hpp"
#include "dependencies/gen.memory.hpp"
#include "dependencies/gen.string_ops.hpp"
#include "dependencies/gen.printing.hpp"
#include "dependencies/gen.containers.hpp"
#include "dependencies/gen.string.hpp"
#include "dependencies/gen.file_handling.hpp"
#include "dependencies/gen.parsing.hpp"
#include "dependencies/gen.timing.hpp"
#include "dependencies/macros.hpp"
#include "dependencies/basic_types.hpp"
#include "dependencies/debug.hpp"
#include "dependencies/memory.hpp"
#include "dependencies/string_ops.hpp"
#include "dependencies/printing.hpp"
#include "dependencies/containers.hpp"
#include "dependencies/hashing.hpp"
#include "dependencies/string.hpp"
#include "dependencies/parsing.hpp"
#include "dependencies/timing.hpp"
#include "dependencies/file_handling.hpp"
GEN_NS_END

22
project/gen.hpp
View File

@ -8,8 +8,8 @@
*/
#pragma once
#include "helpers/gen.push_ignores.inline.hpp"
#include "components/gen.header_start.hpp"
#include "helpers/push_ignores.inline.hpp"
#include "components/header_start.hpp"
#ifdef GEN_DONT_USE_NAMESPACE
# define GEN_NS_BEGIN
@ -21,16 +21,16 @@
GEN_NS_BEGIN
#include "components/gen.types.hpp"
#include "components/gen.ecode.hpp"
#include "components/gen.eoperator.hpp"
#include "components/gen.especifier.hpp"
#include "components/gen.data_structures.hpp"
#include "components/gen.interface.hpp"
#include "components/gen.header_end.hpp"
#include "components/types.hpp"
#include "components/ecode.hpp"
#include "components/eoperator.hpp"
#include "components/especifier.hpp"
#include "components/data_structures.hpp"
#include "components/interface.hpp"
#include "components/header_end.hpp"
#include "filesystem/gen.builder.hpp"
#include "file_processors/builder.hpp"
GEN_NS_END
#include "helpers/gen.pop_ignores.inline.hpp"
#include "helpers/pop_ignores.inline.hpp"

3
project/helpers/gen.helper.hpp → project/helpers/helper.hpp
View File

@ -50,7 +50,6 @@ CodeBody gen_ecode( char const* path )
#pragma pop_macro( "local_persist" )
CodeNamespace nspace = def_namespace( name(ECode), def_namespace_body( args( enum_code, to_str ) ) );
CodeUsing code_t = def_using( name(CodeT), def_type( name(ECode::Type) ) );
return def_global_body( args( nspace, code_t ) );
@ -217,8 +216,6 @@ CodeBody gen_etoktype( char const* etok_path, char const* attr_path )
CSV_Object csv_enum_nodes;
csv_parse( &csv_enum_nodes, rcast(char*, enum_content.data), GlobalAllocator, false );
// memset( scratch_mem, 0, sizeof(scratch_mem) );
// scratch = Arena::init_from_memory( scratch_mem, sizeof(scratch_mem) );
FileContents attrib_content = file_read_contents( scratch, zero_terminate, attr_path );
CSV_Object csv_attr_nodes;

0
project/helpers/gen.pop_ignores.inline.hpp → project/helpers/pop_ignores.inline.hpp
View File

0
project/helpers/gen.push_ignores.inline.hpp → project/helpers/push_ignores.inline.hpp
View File

0
project/helpers/gen.undef.macros.hpp → project/helpers/undef.macros.hpp
View File

8
scripts/.clang-format
View File

@ -3,7 +3,7 @@
AccessModifierOffset: -4
AlignAfterOpenBracket: BlockIndent
AlignArrayOfStructures: Right
AlignArrayOfStructures: Left
AlignConsecutiveAssignments:
Enabled: true
AcrossEmptyLines: false
@ -61,7 +61,7 @@ BraceWrapping:
BeforeLambdaBody: false
BeforeWhile: false
# BreakAfterAttributes: Always
BreakAfterAttributes: Always
# BreakArrays: false
# BreakBeforeInlineASMColon: OnlyMultiline
BreakBeforeBinaryOperators: NonAssignment
@ -96,7 +96,7 @@ IncludeBlocks: Preserve
IndentCaseBlocks: false
IndentCaseLabels: false
IndentExternBlock: AfterExternBlock
IndentGotoLabels: false
IndentGotoLabels: true
IndentPPDirectives: AfterHash
IndentRequires: true
IndentWidth: 4
@ -127,7 +127,7 @@ SeparateDefinitionBlocks: Always
ShortNamespaceLines: 40
SortIncludes: true
SortIncludes: false
SortUsingDeclarations: true
SpaceAfterCStyleCast: false

36
scripts/Readme.md
View File

@ -1,12 +1,34 @@
# Scripts
Build and cleanup scripts for the test directory are found here along with `natvis` and `natstepfilter` files for debugging.
Generation, testing, and cleanup scripts for the test directory are found here along with `natvis` and `natstepfilter` files for debugging.
The build works as follows:
## Refactoring
* Compile and run the meta-program, it will dump files to the `test/gen` directory.
* Format the files using clang-format
* Build a program that uses some the generated definitions. (Have not done yet)
`refactor.ps1` Provides a way to run the [refactor](github.com/Ed94/refactor) program. It uses the `gencpp.refactor` script to complete a mass refactor of all content within the files of the specified within the script.
The `test/gen` directory has the meson.build config for the meta-program
The `test` directory has the one for the dependent-program.
Currently `refactor` only supports naive sort of *find and replace* feature set and will not be able to rename identifiers excluisvely to a specific context (such as only renaming member names of a specific struct, etc).
## Build & Run Scripts
**`clean.ps1`**
Remove any generated content from the repository.
**`bootstrap.ps1`**
Generate a version of gencpp where components are inlined directly to `gen.<hpp/cpp>` and `gen. <hpp/cpp>`
Any heavily preprocessed code is not inlined and are instead generated using the code in the `helpers` directory.
**`singlheader.build.ps1`**
Generate a single-header version of the library where all code that would normally good in the usual four files (see bootstrap) are inlined into a single `gen.hpp` file.
As with the bootstrap, any heavily preprocessed code is not inlined and instead generated with helper code.
**`test.gen.build.ps1`**
Build the metaprogram for generating the test code.
**`test.gen.ps1`**
Build (if not already) the metaprogram for generating test code, then run it to generate code.
**`test.build.ps1`**
Build and run metaprogram, build test program.
**`test.run.ps1`**
Build and run metaprogram, build and run test program.

2
scripts/build.ps1
View File

@ -1,2 +0,0 @@
cls
Invoke-Expression "& $(Join-Path $PSScriptRoot 'build.ci.ps1') $args"

12
scripts/get_sources.ps1
View File

@ -1,12 +0,0 @@
[string[]] $include = 'gen.cpp' #'*.c', '*.cc', '*.cpp'
# [string[]] $exclude =
$path_root = git rev-parse --show-toplevel
$path_proj = Join-Path $path_root project
$files = Get-ChildItem -Recurse -Path $path_proj -Include $include -Exclude $exclude
$sources = $files | Select-Object -ExpandProperty FullName | Resolve-Path -Relative
$sources = $sources.Replace( '\', '/' )
return $sources

0
scripts/build.ci.ps1 → scripts/test.gen_build.ci.ps1
View File

2
scripts/test.gen_build.ps1 Normal file
View File

@ -0,0 +1,2 @@
cls
Invoke-Expression "& $(Join-Path $PSScriptRoot 'test.gen_build.ci.ps1') $args"

0
scripts/gen.ps1 → scripts/test.gen_run.ps1
View File

0
singleheader/components/gen.header_start.hpp → singleheader/components/header_start.hpp
View File

90
singleheader/gen.singleheader.cpp
View File

@ -2,8 +2,8 @@
#define GEN_ENFORCE_STRONG_CODE_TYPES
#define GEN_EXPOSE_BACKEND
#include "gen.cpp"
#include "filesystem/gen.scanner.hpp"
#include "helpers/gen.helper.hpp"
#include "file_processors/scanner.hpp"
#include "helpers/helper.hpp"
using namespace gen;
@ -59,10 +59,10 @@ int gen_main()
#define project_dir "../project/"
Code push_ignores = scan_file( project_dir "helpers/gen.push_ignores.inline.hpp" );
Code pop_ignores = scan_file( project_dir "helpers/gen.pop_ignores.inline.hpp" );
Code push_ignores = scan_file( project_dir "helpers/push_ignores.inline.hpp" );
Code pop_ignores = scan_file( project_dir "helpers/pop_ignores.inline.hpp" );
Code header_start = scan_file( "components/gen.header_start.hpp" );
Code header_start = scan_file( "components/header_start.hpp" );
Code nspace_macro = untyped_str( namespace_by_default ? nspace_default : nspace_non_default );
Builder
@ -80,19 +80,19 @@ int gen_main()
{
header.print_fmt( roll_own_dependencies_guard_start );
Code header_start = scan_file( project_dir "dependencies/gen.header_start.hpp" );
Code macros = scan_file( project_dir "dependencies/gen.macros.hpp" );
Code basic_types = scan_file( project_dir "dependencies/gen.basic_types.hpp" );
Code debug = scan_file( project_dir "dependencies/gen.debug.hpp" );
Code memory = scan_file( project_dir "dependencies/gen.memory.hpp" );
Code string_ops = scan_file( project_dir "dependencies/gen.string_ops.hpp" );
Code printing = scan_file( project_dir "dependencies/gen.printing.hpp" );
Code containers = scan_file( project_dir "dependencies/gen.containers.hpp" );
Code hashing = scan_file( project_dir "dependencies/gen.hashing.hpp" );
Code string = scan_file( project_dir "dependencies/gen.string.hpp" );
Code file_handling = scan_file( project_dir "dependencies/gen.file_handling.hpp" );
Code parsing = scan_file( project_dir "dependencies/gen.parsing.hpp" );
Code timing = scan_file( project_dir "dependencies/gen.timing.hpp" );
Code header_start = scan_file( project_dir "dependencies/header_start.hpp" );
Code macros = scan_file( project_dir "dependencies/macros.hpp" );
Code basic_types = scan_file( project_dir "dependencies/basic_types.hpp" );
Code debug = scan_file( project_dir "dependencies/debug.hpp" );
Code memory = scan_file( project_dir "dependencies/memory.hpp" );
Code string_ops = scan_file( project_dir "dependencies/string_ops.hpp" );
Code printing = scan_file( project_dir "dependencies/printing.hpp" );
Code containers = scan_file( project_dir "dependencies/containers.hpp" );
Code hashing = scan_file( project_dir "dependencies/hashing.hpp" );
Code string = scan_file( project_dir "dependencies/string.hpp" );
Code file_handling = scan_file( project_dir "dependencies/file_handling.hpp" );
Code parsing = scan_file( project_dir "dependencies/parsing.hpp" );
Code timing = scan_file( project_dir "dependencies/timing.hpp" );
header.print( header_start );
header.print_fmt( "GEN_NS_BEGIN\n\n" );
@ -113,16 +113,16 @@ int gen_main()
header.print_fmt( roll_own_dependencies_guard_end );
}
Code types = scan_file( project_dir "components/gen.types.hpp" );
Code data_structs = scan_file( project_dir "components/gen.data_structures.hpp" );
Code interface = scan_file( project_dir "components/gen.interface.hpp" );
Code header_end = scan_file( project_dir "components/gen.header_end.hpp" );
Code types = scan_file( project_dir "components/types.hpp" );
Code data_structs = scan_file( project_dir "components/data_structures.hpp" );
Code interface = scan_file( project_dir "components/interface.hpp" );
Code header_end = scan_file( project_dir "components/header_end.hpp" );
CodeBody ecode = gen_ecode( project_dir "components/ECode.csv" );
CodeBody eoperator = gen_eoperator( project_dir "components/EOperator.csv" );
CodeBody especifier = gen_especifier( project_dir "components/ESpecifier.csv" );
CodeBody ecode = gen_ecode( project_dir "enums/ECode.csv" );
CodeBody eoperator = gen_eoperator( project_dir "enums/EOperator.csv" );
CodeBody especifier = gen_especifier( project_dir "enums/ESpecifier.csv" );
Code builder = scan_file( project_dir "filesystem/gen.builder.hpp" );
Code builder = scan_file( project_dir "file_processors/builder.hpp" );
header.print_fmt( "GEN_NS_BEGIN\n\n" );
@ -146,17 +146,17 @@ int gen_main()
{
header.print_fmt( roll_own_dependencies_guard_start );
Code impl_start = scan_file( project_dir "dependencies/gen.impl_start.cpp" );
Code debug = scan_file( project_dir "dependencies/gen.debug.cpp" );
Code string_ops = scan_file( project_dir "dependencies/gen.string_ops.cpp" );
Code printing = scan_file( project_dir "dependencies/gen.printing.cpp" );
Code memory = scan_file( project_dir "dependencies/gen.memory.cpp" );
Code parsing = scan_file( project_dir "dependencies/gen.parsing.cpp" );
Code hashing = scan_file( project_dir "dependencies/gen.hashing.cpp" );
Code string = scan_file( project_dir "dependencies/gen.string.cpp" );
Code timing = scan_file( project_dir "dependencies/gen.timing.cpp" );
Code impl_start = scan_file( project_dir "dependencies/impl_start.cpp" );
Code debug = scan_file( project_dir "dependencies/debug.cpp" );
Code string_ops = scan_file( project_dir "dependencies/string_ops.cpp" );
Code printing = scan_file( project_dir "dependencies/printing.cpp" );
Code memory = scan_file( project_dir "dependencies/memory.cpp" );
Code parsing = scan_file( project_dir "dependencies/parsing.cpp" );
Code hashing = scan_file( project_dir "dependencies/hashing.cpp" );
Code string = scan_file( project_dir "dependencies/string.cpp" );
Code timing = scan_file( project_dir "dependencies/timing.cpp" );
Code file_handling = scan_file( project_dir "dependencies/gen.file_handling.cpp" );
Code file_handling = scan_file( project_dir "dependencies/file_handling.cpp" );
header.print_fmt( "GEN_NS_BEGIN\n\n");
header.print( impl_start );
@ -176,18 +176,18 @@ int gen_main()
header.print_fmt( roll_own_dependencies_guard_end );
}
Code data = scan_file( project_dir "components/gen.data.cpp" );
Code ast_case_macros = scan_file( project_dir "components/gen.ast_case_macros.cpp" );
Code ast = scan_file( project_dir "components/gen.ast.cpp" );
Code interface = scan_file( project_dir "components/gen.interface.cpp" );
Code upfront = scan_file( project_dir "components/gen.interface.upfront.cpp" );
Code parsing = scan_file( project_dir "components/gen.interface.parsing.cpp" );
Code untyped = scan_file( project_dir "components/gen.untyped.cpp" );
Code data = scan_file( project_dir "components/static_data.cpp" );
Code ast_case_macros = scan_file( project_dir "components/ast_case_macros.cpp" );
Code ast = scan_file( project_dir "components/ast.cpp" );
Code interface = scan_file( project_dir "components/interface.cpp" );
Code upfront = scan_file( project_dir "components/interface.upfront.cpp" );
Code parsing = scan_file( project_dir "components/interface.parsing.cpp" );
Code untyped = scan_file( project_dir "components/untyped.cpp" );
CodeBody etoktype = gen_etoktype( project_dir "components/ETokType.csv", project_dir "components/AttributeTokens.csv" );
CodeBody etoktype = gen_etoktype( project_dir "enums/ETokType.csv", project_dir "enums/AttributeTokens.csv" );
CodeNamespace parser_nspace = def_namespace( name(Parser), def_namespace_body( args(etoktype)) );
Code builder = scan_file( project_dir "filesystem/gen.builder.cpp" );
Code builder = scan_file( project_dir "file_processors/builder.cpp" );
header.print_fmt( "GEN_NS_BEGIN\n\n");
header.print( data );

2
test/Parsed/Array.Parsed.hpp → test/parsed/Array.Parsed.hpp
View File

@ -229,7 +229,7 @@ Array<GenArrayRequest> GenArrayRequests;
void gen__array_request( StrC type, StrC dep = {} )
{
do_once_start
GenArrayRequests = Array<GenArrayRequest>::init( Memory::GlobalAllocator );
GenArrayRequests = Array<GenArrayRequest>::init( GlobalAllocator );
do_once_end
// Make sure we don't already have a request for the type.

2
test/Parsed/Buffer.Parsed.hpp → test/parsed/Buffer.Parsed.hpp
View File

@ -143,7 +143,7 @@ Array<GenBufferRequest> GenBufferRequests;
void gen__buffer_request( StrC type, StrC dep = {} )
{
do_once_start
GenBufferRequests = Array<GenBufferRequest>::init( Memory::GlobalAllocator );
GenBufferRequests = Array<GenBufferRequest>::init( GlobalAllocator );
do_once_end
// Make sure we don't already have a request for the type.

2
test/Parsed/HashTable.Parsed.hpp → test/parsed/HashTable.Parsed.hpp
View File

@ -292,7 +292,7 @@ Array<GenHashTableRequest> GenHashTableRequests;
void gen__hashtable_request( StrC type, StrC dep = {} )
{
do_once_start
GenHashTableRequests = Array<GenHashTableRequest>::init( Memory::GlobalAllocator );
GenHashTableRequests = Array<GenHashTableRequest>::init( GlobalAllocator );
gen_array( sw );
do_once_end

2
test/Parsed/Ring.Parsed.hpp → test/parsed/Ring.Parsed.hpp
View File

@ -109,7 +109,7 @@ Array<GenRingRequest> GenRingRequests;
void gen__ring_request( StrC type, StrC dep = {} )
{
do_once_start
GenRingRequests = Array<GenRingRequest>::init( Memory::GlobalAllocator );
GenRingRequests = Array<GenRingRequest>::init( GlobalAllocator );
do_once_end
// Make sure we don't already have a request for the type.

0
test/Parsed/Sanity.Parsed.hpp → test/parsed/Sanity.Parsed.hpp
View File

0
test/test.Upfront.cpp → test/test.upfront.cpp
View File

4
test/Upfront/Array.Upfront.hpp → test/upfront/Array.Upfront.hpp
View File

@ -1,5 +1,7 @@
#pragma once
#if GEN_TIME
#include "gen.hpp"
@ -308,7 +310,7 @@ Array<GenArrayRequest> GenArrayRequests;
void gen__array_request( StrC type, StrC dep = {} )
{
do_once_start
GenArrayRequests = Array<GenArrayRequest>::init( Memory::GlobalAllocator );
GenArrayRequests = Array<GenArrayRequest>::init( GlobalAllocator );
do_once_end
// Make sure we don't already have a request for the type.

3
test/Upfront/Buffer.Upfront.hpp → test/upfront/Buffer.Upfront.hpp
View File

@ -190,6 +190,7 @@ Code gen__buffer( StrC type, sw type_size )
, free
, get_header
, num
, pop
, wipe
, op_type_ptr
@ -212,7 +213,7 @@ Array<GenBufferRequest> GenBufferRequests;
void gen__buffer_request( StrC type, StrC dep = {} )
{
do_once_start
GenBufferRequests = Array<GenBufferRequest>::init( Memory::GlobalAllocator );
GenBufferRequests = Array<GenBufferRequest>::init( GlobalAllocator );
do_once_end
// Make sure we don't already have a request for the type.

2
test/Upfront/HashTable.Upfront.hpp → test/upfront/HashTable.Upfront.hpp
View File

@ -419,7 +419,7 @@ Array<GenHashTableRequest> GenHashTableRequests;
void gen__hashtable_request( StrC type, StrC dep = {} )
{
do_once_start
GenHashTableRequests = Array<GenHashTableRequest>::init( Memory::GlobalAllocator );
GenHashTableRequests = Array<GenHashTableRequest>::init( GlobalAllocator );
gen_array( sw );
do_once_end

2
test/Upfront/Ring.Upfront.hpp → test/upfront/Ring.Upfront.hpp
View File

@ -163,7 +163,7 @@ Array<GenRingRequest> GenRingRequests;
void gen__ring_request( StrC type, StrC dep = {} )
{
do_once_start
GenRingRequests = Array<GenRingRequest>::init( Memory::GlobalAllocator );
GenRingRequests = Array<GenRingRequest>::init( GlobalAllocator );
do_once_end
// Make sure we don't already have a request for the type.

2
test/Upfront/Sanity.Upfront.hpp → test/upfront/Sanity.Upfront.hpp
View File

@ -287,7 +287,7 @@ u32 gen_sanity_upfront()
// Using
{
CodeUsing reg = def_using( name(TestUsing), t_u8 );
CodeUsingNamespace nspace = def_using_namespace( name(TestNamespace) );
CodeUsing nspace = def_using_namespace( name(TestNamespace) );
gen_sanity_file.print(reg);
gen_sanity_file.print(nspace);