12 KiB
Parser's Algorithim
gencpp uses a hand-written recursive descent parser. Both the lexer and parser currently handle a full C/C++ file in a single pass.
Notable implementation background
Lexer
The lex procedure does the lexical pass of content provided as a StrC
type.
The tokens are stored (for now) in gen::parser::Tokens
.
Fields:
Array<Token> Arr;
s32 Idx;
What token types are supported can be found in ETokType.csv you can also find the token types in ETokType.h , which is the generated enum from the csv file.
Tokens are defined with the struct gen::parser::Token
:
Fields:
char const* Text;
sptr Length;
TokType Type;
s32 Line;
s32 Column;
u32 Flags;
Flags is a bitfield made up of TokFlags (Token Flags):
TF_Operator
: Any operator token used in expressionsTF_Assign
- Using statment assignment
- Parameter argument default value assignment
- Variable declaration initialization assignment
TF_Preprocess
: Related to a preprocessing directiveTF_Preprocess_Cond
: A preprocess conditionalTF_Attribute
: An attribute tokenTF_AccessSpecifier
: An accesor operation tokenTF_Specifier
: One of the specifier tokensTF_EndDefinition
: Can be interpreted as an end definition for a scope.TF_Formatting
: Considered a part of the formattingTF_Literal
: Anything considered a literal by C++.
I plan to replace IsAssign with a general flags field and properly keep track of all operator types instead of abstracting it away to ETokType::Operator
.
Traversing the tokens is done with the following interface macros:
Macro | Description |
---|---|
currtok_noskip |
Get the current token without skipping whitespace |
currtok |
Get the current token, skip any whitespace tokens |
prevtok |
Get the previous token (does not skip whitespace) |
nexttok |
Get the next token (does not skip whitespace) |
eat( Token Type ) |
Check to see if the current token is of the given type, if so, advance Token's index to the next token |
left |
Get the number of tokens left in the token array |
check_noskip |
Check to see if the current token is of the given type, without skipping whitespace |
check |
Check to see if the current token is of the given type, skip any whitespace tokens |
Parser
The parser has a limited user interface, only specific types of definitions or statements are expected to be provided by the user directly when using to construct an AST dynamically (See SOA for example). It however does attempt to provide capability to parse a full C/C++ from production codebases.
Each public user interface procedure has the following format:
<code type> parse_<definition type>( StrC def )
{
check_parse_args( def );
using namespace Parser;
TokArray toks = lex( def );
if ( toks.Arr == nullptr )
return CodeInvalid;
// Parse the tokens and return a constructed AST using internal procedures
...
}
The most top-level parsing procedure used for C/C++ file parsing is parse_global_body
:
It uses a helper procedure called parse_global_nspace
.
Each internal procedure will have the following format:
internal
<code type> parse_<definition_type>( <empty or contextual params> )
{
push_scope();
...
<code type> result = (<code type>) make_code();
...
Context.pop();
return result;
}
Below is an outline of the general alogirithim used for these internal procedures. The intention is to provide a basic briefing to aid the user in traversing the actual code definitions. These appear in the same order as they are in the parser.cpp
file
parse_array_decl
- Check if its an array declaration with no expression.
- Consume and return empty array declaration
- Opening square bracket
- Consume expression
- Closing square bracket
- If adjacent opening bracket
- Repeat array declaration parse until no brackets remain
parse_attributes
- Check for standard attribute
- Check for GNU attribute
- Check for MSVC attribute
- Check for a token registered as an attribute
parse_class_struct
- Check for export module specifier
- class or struct keyword
parse_attributes
- If identifier :
parse_identifier
- Parse inherited parent or interfaces
- If opening curly brace :
parse_class_struct_body
- If not an inplace definition
- End statement
- Check for inline comment
parse_class_struct_body
- Opening curly brace
- Parse the body (Possible options):
- Newline : ast constant
- Comment :
parse_comment
- Access_Public : ast constant
- Access_Protected : ast constant
- Access_Private : ast constant
- Decl_Class :
parse_complicated_definition
- Decl_Enum :
parse_complicated_definition
- Decl_Friend :
parse_friend
- Decl_Operator :
parse_operator_cast
- Decl_Struct :
parse_complicated_definition
- Decl_Template :
parse_template
- Decl_Typedef :
parse_typedef
- Decl_Union :
parse_complicated_definition
- Decl_Using :
parse_using
- Operator == '~'
parse_destructor
- Preprocess_Define :
parse_define
- Preprocess_Include :
parse_include
- Preprocess_Conditional (if, ifdef, ifndef, elif, else, endif) :
parse_preprocess_cond
or else/endif ast constant - Preprocess_Macro :
parse_simple_preprocess
- Preprocess_Pragma :
parse_pragma
- Preprocess_Unsupported :
parse_simple_preprocess
- StaticAssert :
parse_static_assert
- The following compound into a resolved definition or declaration:
- Attributes (Standard, GNU, MSVC) :
parse_attributes
- Specifiers (consteval, constexpr, constinit, forceinline, inline, mutable, neverinline, static, volatile)
- Possible Destructor :
parse_destructor
- Possible User defined operator cast :
parse_operator_cast
- Possible Constructor :
parse_constructor
- Something that has the following: (identifier, const, unsigned, signed, short, long, bool, char, int, double)
- Possible Constructor
parse_constructor
- Possible Operator, Function, or varaible :
parse_operator_function_or_variable
- Possible Constructor
- Attributes (Standard, GNU, MSVC) :
- Something completely unknown (will just make untyped...) :
parse_untyped
parse_comment
- Just wrap the token into a cached string ( the lexer did the processing )
parse_compilcated_definition
This is a helper function used by the following functions to help resolve a declaration or definition:
parse_class_struct_body
parse_global_nspace
parse_union
A portion of the code in parse_typedef
is very similar to this as both have to resolve a similar issue.
- Look ahead to the termination token (End statement)
- Check to see if it fits the pattern for a forward declare
- If the previous token was an identifier (
token[-1]
):- Look back one more token :
[-2]
- If the token has a closing brace its an inplace definition
- If the
token[-2]
is an identifier &token[-3]
is the declaration type, its a variable using a namespaced type. - If the
token[-2]
is an indirection, then its a variable using a namespaced/forwarded type. - If any of the above is the case,
parse_operator_function_or_variable
- Look back one more token :
- If the previous token was a closing curly brace, its a definition :
parse_forward_or_definition
- If the previous token was a closing square brace, its an array definition :
parse_operator_function_or_variable
parse_define
- Define directive
- Get identifier
- Get Content
parse_forward_or_definition
- Parse any of the following for either a forward declaration or definition:
- Decl_Class :
parse_class
- Decl_Enum :
parse_enum
- Decl_Struct :
parse_struct
- Decl_Union :
parse_union
- Decl_Class :
parse_function_after_name
This is needed as a function defintion is not easily resolvable early on, as such this function handles resolving a function after its been made ceratin that the type of declaration or definition is indeed for a function signature.
By the point this function is called the following are known : export module flag, attributes, specifiers, return type, & name
parse_parameters
- parse postfix specifiers (we do not check if the specifier here is correct or not to be here... yet)
- If there is a body :
parse_body
- Otherwise :
- Statment end
- Check for inline comment
parse_function_body
Currently there is no actual parsing of the function body. Any content with the braces is shoved into an execution AST node. In the future statements and expressions will be parsed.
- Open curly brace
- Grab all tokens between the brace and the closing brace, shove them in a execution AST node.
- Closing curly brace
parse_global_nspace
- Make sure this is being called for a valid type (namespace, global body, export body, linkage body)
- If its not a global body, consume the opening curly brace
- Parse the body (Possible options):
- NewLine : ast constant
- Comment :
parse_comment
- Decl_Cass :
parse_complicated_definition
- Decl_Enum :
parse_complicated_definition
- Decl_Extern_Linkage :
parse_extern_link
- Decl_Namespace :
parse_namespace
- Decl_Struct :
parse_complicated_definition
- Decl_Template :
parse_template
- Decl_Typedef :
parse_typedef
- Decl_Union :
parse_complicated_definition
- Decl_Using :
parse_using
- Preprocess_Define :
parse_define
- Preprocess_Include :
parse_include
- Preprocess_If, IfDef, IfNotDef, Elif :
parse_preprocess_cond
- Preprocess_Else : ast constant
- Preprocess_Endif : ast constant
- Preprocess_Macro :
parse_simple_preprocess
- Preprocess_Pragma :
parse_pragma
- Preprocess_Unsupported :
parse_simple_preprocess
- StaticAssert :
parse_static_assert
- Module_Export :
parse_export_body
- Module_Import : NOT_IMPLEMENTED
- The following compound into a resolved definition or declaration:
- Attributes ( Standard, GNU, MSVC, Macro ) :
parse_attributes
- Specifiers ( consteval, constexpr, constinit, extern, forceinline, global, inline, internal_linkage, neverinline, static )
- Is either ( identifier, const specifier, long, short, signed, unsigned, bool, char, double, int)
- If its an operator cast (definition outside class) :
parse_operator_cast
- Its an operator, function, or varaible :
parse_operator_function_or_varaible
- If its an operator cast (definition outside class) :
- Attributes ( Standard, GNU, MSVC, Macro ) :
- If its not a global body, consuem the closing curly brace
parse_identifier
parse_include
parse_operator_after_ret_type
parse_operator_function_or_variable
parse_pragma
parse_params
parse_preprocess_cond
parse_simple_preprocess
parse_static_assert
parse_template_args
parse_variable_after_name
parse_variable_declaration_list
parse_class
parse_constructor
parse_destructor
parse_enum
parse_export_body
parse_extern_link_body
parse_extern_link
parse_friend
parse_function
parse_namespace
parse_operator
parse_operator_cast
parse_struct
parse_template
parse_type
parse_typedef
- Check for export module specifier
- typedef keyword
- If its a preprocess macro: Get the macro name
parse_union
- Check for export module specifier
- union keyword
parse_attributes
- Check for identifier
- Parse the body (Possible options):
- Newline
- Comment
- Decl_Class
- Decl_Enum
- Decl_Struct
- Decl_Union
- Preprocess_Define
- Preprocess_Conditional (if, ifdef, ifndef, elif, else, endif)
- Preprocess_Macro
- Preprocess_Pragma
- Unsupported preprocess directive
- Variable
- If its not an inplace definiton: End Statement
parse_using
- Check for export module specifier
- using keyword
- Check to see if its a using namespace
- Get the identifier
- If its a regular using declaration:
parse_attributes
parse_type
parse_array_decl
- End statement
- Check for inline comment
parse_variable
- Check for export module specifier
parse_attributes
parse specifiers
parse_type
parse_identifier
parse_variable_after_name