codebase readme pass

README.md

@@ -17,7 +17,7 @@ with any information you can gather, like dump files (along with the build you
 used), instructions to reproduce, test executables, and so on.
 
 You can download pre-built binaries for the debugger
-[here](https://github.com/EpicGames/raddebugger/releases).
+[here](https://github.com/EpicGamesExt/raddebugger/releases).
 
 The RAD Debugger project aims to simplify the debugger by simplifying and
 unifying the underlying debug info format. In that pursuit we've built the RAD
@@ -90,13 +90,12 @@ You should see the following output:
 ```
 [debug mode]
 [msvc compile]
-[default mode, assuming `raddbg` build]
 metagen_main.c
-searching C:\devel\raddebugger/src... 299 files found
+searching C:\devel\raddebugger/src... 309 files found
-parsing metadesk... 12 metadesk files parsed
+parsing metadesk... 15 metadesk files parsed
-gathering tables... 37 tables found
+gathering tables... 96 tables found
 generating layer code...
-raddbg.cpp
+raddbg_main.c
 ```
 
 If everything worked correctly, there will be a `build` folder in the root
@@ -169,7 +168,8 @@ After setting up the codebase and building, the following directories will
 also exist:
 
 - `build`: All build artifacts. Not checked in to version control.
-- `local`: Local files, used for local build configuration input files.
+- `local`: Local files, used for local build configuration input files. Not
+  checked in to version control.
 
 ## Codebase Introduction
 
@@ -213,55 +213,70 @@ A list of the layers in the codebase and their associated namespaces is below:
   processes. Runs in lockstep with attached processes. When it runs, attached
   processes are halted. When attached processes are running, it is halted.
   Driven by a debugger frontend on another thread.
-- `dasm` (`DASM_`): An asynchronous disassembly decoder and cache. Users ask for
+- `dasm_cache` (`DASM_`): An asynchronous disassembly decoder and cache. Users
-  disassembly for a particular virtual address range in a process, and threads
+  ask for disassembly for some data, with a particular architecture, and other
-  implemented in this layer decode and cache the disassembly for that range.
+  various parameters, and threads implemented in this layer decode and cache the
+  disassembly for that data with those parameters.
 - `dbgi` (`DI_`): An asynchronous debug info loader and cache. Loads debug info
   stored in the RDI format. Users ask for debug info for a particular path, and
   on separate threads, this layer loads the associated debug info file. If
   necessary, it will launch a separate conversion process to convert original
   debug info into the RDI format.
-- `demon` (`DEMON_`): An abstraction layer for local-machine, low-level process
+- `dbg_engine` (`D_`): Implements the core debugger system, without any
+  graphical components. This contains top-level logic for things like stepping,
+  launching, freezing threads, mid-run breakpoint addition, some caching layers,
+  and so on.
+- `demon` (`DMN_`): An abstraction layer for local-machine, low-level process
   control. The abstraction is used to provide a common interface for process
   control on target platforms. Used to implement part of `ctrl`.
-- `df/core` (`DF_`): The debugger's non-graphical frontend. Implements a
+- `draw` (`DR_`): Implements a high-level graphics drawing API for the
-  debugger "entity cache" (where "entities" include processes, threads, modules,
+  debugger's purposes, using the underlying `render` abstraction layer. Provides
-  breakpoints, source files, targets, and so on). Implements a command loop
+  high-level APIs for various draw commands, but takes care of batching them,
-  for driving process control, which is used to implement stepping commands and
+  and so on.
-  user breakpoints. Implements extractors and caches for various entity-related
+- `eval` (`E_`): Implements a compiler for an expression language built for
-  data, like full thread unwinds and local variable maps. Also implements core
+  evaluation of variables, registers, types, and more, from debugger-attached
-  building blocks for evaluation and evaluation visualization.
+  processes, debug info, debugger state, and files. Broken into several phases
-- `df/gfx` (`DF_`): The debugger's graphical frontend. Builds on top of
+  mostly corresponding to traditional compiler phases - lexer, parser,
-  `df/core` to provide all graphical features, including windows, panels, all
+  type-checker, IR generation, and IR evaluation.
-  of the various debugger interfaces, and evaluation visualization.
+- `eval_visualization` (`EV_`): Implements the core non-graphical evaluation
-- `draw` (`D_`): Implements a high-level graphics drawing API for the debugger's
+  visualization engine, which can be used to visualize evaluations (provided by
-  purposes, using the underlying `render` abstraction layer. Provides high-level
+  the `eval` layer) in a number of ways. Implements core data structures and
-  APIs for various draw commands, but takes care of batching them, and so on.
+  transforms for the `Watch` view.
-- `eval` (`EVAL_`): Implements a compiler for an expression language built for
+- `file_stream` (`FS_`): Provides asynchronous file loading, storing the
-  evaluation of variables, registers, and so on from debugger-attached processes
+  artifacts inside of the cache implemented by the `hash_store` layer, and
-  and/or debug info. Broken into several phases mostly corresponding to
+  hot-reloading the contents of files when they change. Allows callers to map
-  traditional compiler phases - lexer, parser, type-checker, IR generation, and
+  file paths to data hashes, which can then be used to obtain the file's data.
-  IR evaluation.
+- `font_cache` (`FNT_`): Implements a cache of rasterized font data, both in
-- `font_cache` (`F_`): Implements a cache of rasterized font data, both in CPU-
+  CPU-side data for text shaping, and in GPU texture atlases for rasterized
-  side data for text shaping, and in GPU texture atlases for rasterized glyphs.
+  glyphs. All cache information is sourced from the `font_provider` abstraction
-  All cache information is sourced from the `font_provider` abstraction layer.
+  layer.
 - `font_provider` (`FP_`): An abstraction layer for various font file decoding
   and font rasterization backends.
+- `fuzzy_search` (`FZY_`): Provides a fuzzy searching engine for doing
+  large, asynchronous fuzzy searches. Used by the debugger for implementing
+  things like the symbol lister or the `Procedures` view, which search across
+  all loaded debug info records, using fuzzy matching rules.
 - `geo_cache` (`GEO_`): Implements an asynchronously-filled cache for GPU
   geometry data, filled by data sourced in the `hash_store` layer's cache. Used
-  for asynchronously preparing data for memory visualization in the debugger.
+  for asynchronously preparing data for visualization.
 - `hash_store` (`HS_`): Implements a cache for general data blobs, keyed by a
-  128-bit hash of the data. Used as a general data store by other layers.
+  128-bit hash of the data. Also implements a 128-bit key cache on top, where
+  the keys refer to a unique identity, associated with a 128-bit hash, where the
+  hash may change across time. Used as a general data store by other layers.
 - `lib_raddbg_markup` (`RADDBG_`): Standalone library for marking up user
-  programs to work with various features in the `raddbg` debugger. Does not
+  programs to work with various features in the debugger. Does not depend on
-  depend on `base`, and can be independently relocated to other codebases.
+  `base`, and can be independently relocated to other codebases.
-- `lib_rdi_make` (`RDIM_`): Standalone library for constructing RDI debug info
-  data. Does not depend on `base`, and can be independently relocated
-  to other codebases.
 - `lib_rdi_format` (`RDI_`): Standalone library which defines the core RDI types
   and helper functions for reading and writing the RDI debug info file format.
   Does not depend on `base`, and can be independently relocated to other
   codebases.
+- `lib_rdi_make` (`RDIM_`): Standalone library for constructing RDI debug info
+  data. Does not depend on `base`, and can be independently relocated
+  to other codebases.
+- `mdesk` (`MD_`): Code for parsing Metadesk files (stored as `.mdesk`), which
+  is the JSON-like (technically a JSON superset) text format used for the
+  debugger's user and project configuration files, view rules, and metacode,
+  which is parsed and used to generate code with the `metagen` layer.
 - `metagen` (`MG_`): A metaprogram which is used to generate primarily code and
   data tables. Consumes Metadesk files, stored with the extension `.mdesk`, and
   generates C code which is then included by hand-written C code. Currently, it
@@ -279,6 +294,9 @@ A list of the layers in the codebase and their associated namespaces is below:
 - `msf` (`MSF_`): Code for parsing and/or writing the MSF file format.
 - `mule` (no namespace): Test executables for battle testing debugger
   functionality.
+- `mutable_text` (`MTX_`): Implements an asynchronously-filled-and-mutated
+  cache for text buffers which are mutated across time. In the debugger, this is
+  used to implement the `Output` view.
 - `natvis` (no namespace): NatVis files for type visualization of the codebase's
   types in other debuggers.
 - `os/core` (`OS_`): An abstraction layer providing core, non-graphical
@@ -287,20 +305,28 @@ A list of the layers in the codebase and their associated namespaces is below:
 - `os/gfx` (`OS_`): An abstraction layer, building on `os/core`, providing
   graphical operating system features under an abstract API, which is
   implemented per-target-operating-system.
-- `os/socket` (`OS_`): An abstraction layer, building on `os/core`, providing
+- `path` (`PATH_`): Small helpers for manipulating file path strings.
-  networking operating system features under an abstract API, which is
-  implemented per-target-operating-system.
 - `pdb` (`PDB_`): Code for parsing and/or writing the PDB file format.
 - `pe` (`PE_`): Code for parsing and/or writing the PE (Portable Executable)
   file format.
-- `raddbg` (no namespace): The layer which ties everything together for the main
+- `raddbg` (`RD_`): The layer which ties everything together for the main
-  graphical debugger. Not much "meat", just drives `df`, implements command line
+  graphical debugger. Implements the debugger's graphical frontend, all of the
-  options, and so on.
+  debugger-specific UI, the debugger executable's command line interface, and
-- `rdi_from_pdb` (`P2R_`): Our implementation of PDB-to-RDI conversion.
+  all of the built-in visualizers.
-- `rdi_from_dwarf` (`D2R_`): Our in-progress implementation of DWARF-to-RDI
+- `rdi_breakpad_from_pdb` (`P2B_`): Our implementation, using the codebase's RDI
-  conversion.
+  technology, for extracting information from PDBs and generating Breakpad text
+  dumps.
 - `rdi_dump` (no namespace): A dumper utility program for dumping
   textualizations of RDI debug info files.
+- `rdi_format` (no namespace): A layer which includes the `lib_rdi_format` layer
+  and bundles it with codebase-specific helpers, to easily include the library
+  in codebase programs, and have it be integrated with codebase constructs.
+- `rdi_from_dwarf` (`D2R_`): Our in-progress implementation of DWARF-to-RDI
+  conversion.
+- `rdi_from_pdb` (`P2R_`): Our implementation of PDB-to-RDI conversion.
+- `rdi_make` (no namespace): A layer which includes the `lib_rdi_make` layer and
+  bundles it with codebase-specific helpers, to easily include the library in
+  codebase programs, and have it be integrated with codebase constructs.
 - `regs` (`REGS_`): Types, helper functions, and metadata for registers on
   supported architectures. Used in reading/writing registers in `demon`, or in
   looking up register metadata.
@@ -309,19 +335,19 @@ A list of the layers in the codebase and their associated namespaces is below:
   level drawing API - this layer is strictly for minimally abstracting on an
   as-needed basis. Higher level drawing features are implemented in the `draw`
   layer.
-- `scratch` (no namespace): Scratch space for small and transient test or sample
+- `scratch` (no namespace): Scratch space for small and transient test programs.
-  programs.
+- `task_system` (`TS_`): Implements a system for kicking off asynchronous tasks
+  and obtaining artifacts from those tasks once they've completed.
 - `texture_cache` (`TEX_`): Implements an asynchronously-filled cache for GPU
   texture data, filled by data sourced in the `hash_store` layer's cache. Used
-  for asynchronously preparing data for memory visualization in the debugger.
+  for asynchronously preparing data for visualization.
-- `txti` (`TXTI_`): Machinery for asynchronously-loaded, asynchronously hot-
+- `text_cache` (`TXT_`): Implements an asynchronously-filled cache for textual
-  reloaded, asynchronously parsed, and asynchronously mutated source code files.
+  analysis data (tokens, line ranges, and so on), filled by data sourced in the
-  Used by the debugger to visualize source code files. Users ask for text lines,
+  `hash_store` layer's cache. Used for asynchronously preparing data for
-  tokens, and metadata, and it is prepared on background threads.
+  visualization (like for the source code viewer).
-- `type_graph` (`TG_`): Code for analyzing and navigating type structures from
+- `third_party` (no namespace): External code from other projects, which some
-  RDI debug info files, with the additional capability of constructing
+  layers in the codebase depend on. All external code is included and built
-  synthetic types *not* found in debug info. Used in `eval` and for various
+  directly within the codebase.
-  visualization features.
 - `ui` (`UI_`): Machinery for building graphical user interfaces. Provides a
   core immediate mode hierarchical user interface data structure building
   API, and has helper layers for building some higher-level widgets.

src/ico/ico.c

@@ -1,2 +0,0 @@
-// Copyright (c) 2024 Epic Games Tools
-// Licensed under the MIT license (https://opensource.org/license/mit/)

src/ico/ico.h

@@ -1,43 +0,0 @@
-// Copyright (c) 2024 Epic Games Tools
-// Licensed under the MIT license (https://opensource.org/license/mit/)
-
-#ifndef ICO_H
-#define ICO_H
-
-////////////////////////////////
-//~ rjf: ICO File Format Types
-
-#pragma pack(push, 1)
-
-typedef struct ICO_Header ICO_Header;
-struct ICO_Header
-{
-  U16 reserved_padding; // must be 0
-  U16 image_type; // if 1 -> ICO, if 2 -> CUR
-  U16 num_images;
-};
-
-typedef struct ICO_Entry ICO_Entry;
-struct ICO_Entry
-{
-  U8 image_width_px;
-  U8 image_height_px;
-  U8 num_colors;
-  U8 reserved_padding; // should be 0
-  union
-  {
-    U16 ico_color_planes; // in ICO
-    U16 cur_hotspot_x_px; // in CUR
-  };
-  union
-  {
-    U16 ico_bits_per_pixel; // in ICO
-    U16 cur_hotspot_y_px;   // in CUR
-  };
-  U32 image_data_size;
-  U32 image_data_off;
-};
-
-#pragma pack(pop)
-
-#endif // ICO_H

src/raddbg/raddbg_core.c

@@ -10609,6 +10609,35 @@ rd_init(CmdLine *cmdln)
     U8 *opl = ptr+data.size;
     
     // rjf: read header
+#pragma pack(push, 1)
+    typedef struct ICO_Header ICO_Header;
+    struct ICO_Header
+    {
+      U16 reserved_padding; // must be 0
+      U16 image_type; // if 1 -> ICO, if 2 -> CUR
+      U16 num_images;
+    };
+    typedef struct ICO_Entry ICO_Entry;
+    struct ICO_Entry
+    {
+      U8 image_width_px;
+      U8 image_height_px;
+      U8 num_colors;
+      U8 reserved_padding; // should be 0
+      union
+      {
+        U16 ico_color_planes; // in ICO
+        U16 cur_hotspot_x_px; // in CUR
+      };
+      union
+      {
+        U16 ico_bits_per_pixel; // in ICO
+        U16 cur_hotspot_y_px;   // in CUR
+      };
+      U32 image_data_size;
+      U32 image_data_off;
+    };
+#pragma pack(pop)
     ICO_Header hdr = {0};
     if(ptr+sizeof(hdr) < opl)
     {

src/raddbg/raddbg_main.c

@@ -538,7 +538,6 @@
 #include "base/base_inc.h"
 #include "os/os_inc.h"
 #include "task_system/task_system.h"
-#include "ico/ico.h"
 #include "rdi_format/rdi_format_local.h"
 #include "rdi_make/rdi_make_local.h"
 #include "mdesk/mdesk.h"
@@ -578,7 +577,6 @@
 #include "base/base_inc.c"
 #include "os/os_inc.c"
 #include "task_system/task_system.c"
-#include "ico/ico.c"
 #include "rdi_format/rdi_format_local.c"
 #include "rdi_make/rdi_make_local.c"
 #include "mdesk/mdesk.c"