diff --git a/docs/reports/TEST_REGRESSION_ANALYSIS_MINIMAX_OPENAI_20260613.md b/docs/reports/TEST_REGRESSION_ANALYSIS_MINIMAX_OPENAI_20260613.md new file mode 100644 index 00000000..8562921e --- /dev/null +++ b/docs/reports/TEST_REGRESSION_ANALYSIS_MINIMAX_OPENAI_20260613.md @@ -0,0 +1,55 @@ +# Test Regression Analysis: MiniMax & OpenAI Compatible Senders + +**Track/Context:** Hardening & Verification +**Date:** 2026-06-13 +**Status:** Completed +**Subject:** Analysis of why previous tests missed the MiniMax/OpenAI compatible client regressions and details on how we hardened the tests to prevent them. + +--- + +## 1. Why the Tests Missed the Regressions + +The recent regressions with the MiniMax / OpenAI-compatible shims (including mismatched international/domestic base URLs causing 401s, NameErrors, and type mismatches on the new `Result` wrapper) were missed by the test suite due to three primary testing gaps: + +### Gap 1: Total Mocking of Initialization Functions +In the existing tests, the initialization routines (such as `_ensure_minimax_client()` or `_ensure_grok_client()`) were patched out entirely using `unittest.mock.patch`: +```python +with patch("src.ai_client._ensure_minimax_client", return_value=MagicMock()): + ... +``` +By replacing the entire initialization function with a dummy mock, the test suite **never executed the actual code** inside those functions. Consequently: +* The name error when importing modules was bypassed. +* The lookup of base URLs and API keys in `credentials.toml` was skipped. +* The parameters passed to `openai.OpenAI()` were never validated. + +### Gap 2: Lack of Unit Tests for `_ensure_*_client` +There were no unit tests focused specifically on client setup/instantiation logic under different credential profiles. The tests only verified what happens *after* a client is assumed to be successfully instantiated. + +### Gap 3: Mismatched Mock Return Types +The test mocks returned raw strings or custom mock responses, rather than validating that the helper senders correctly return a `Result[str]` wrapper (resulting in assertions like `assert result == "hi from grok"` failing because the sender returned `Result(data="hi from grok")`). + +--- + +## 2. Test Hardening Steps Taken + +To ensure these regressions are caught automatically in the future, we have implemented the following unit tests in [tests/test_minimax_provider.py](file:///C:/projects/manual_slop/tests/test_minimax_provider.py): + +1. **`test_minimax_ensure_client_instantiation`**: + * Patches out credentials loading and the warmed `openai` module. + * Invokes `_ensure_minimax_client()` directly to force instantiation. + * Asserts that `openai.OpenAI()` is called with the exact expected credentials and base URL (e.g. `base_url="https://api.minimax.io/v1"`). +2. **`test_minimax_ensure_client_missing_key_raises_value_error`**: + * Simulates a missing API key in `credentials.toml`. + * Asserts that `_ensure_minimax_client()` raises a `ValueError` with the message `"MiniMax API key not found in credentials.toml"`. + +Both tests run on a clean global client state and execute the actual implementation code rather than mocking it away. + +--- + +## 3. Future Recommendations for Other Providers + +To achieve complete coverage and prevent similar regressions across all other OpenAI-compatible shims (Grok, Llama, Qwen, DeepSeek): + +* **Implement Instantiation Tests**: Replicate the `test_minimax_ensure_client_instantiation` pattern for `_ensure_grok_client()`, `_ensure_llama_client()`, `_ensure_qwen_client()`, and `_ensure_deepseek_client()`. +* **Validate Credentials Handling**: Test each provider's error response when keys are missing or invalid in `credentials.toml`. +* **Type-Check Return Values**: Ensure all provider integration tests assert that their returns match the `Result[str]` type, accessing the response text via `result.data`.