docs(reports): add MiniMax & OpenAI test regression analysis report

2026-06-13 18:57:56 -04:00
parent 2e91cd7123
commit c87bb46e4f
1 changed files with 55 additions and 0 deletions
@@ -0,0 +1,55 @@
+# Test Regression Analysis: MiniMax & OpenAI Compatible Senders
+
+**Track/Context:** Hardening & Verification  
+**Date:** 2026-06-13  
+**Status:** Completed  
+**Subject:** Analysis of why previous tests missed the MiniMax/OpenAI compatible client regressions and details on how we hardened the tests to prevent them.
+
+---
+
+## 1. Why the Tests Missed the Regressions
+
+The recent regressions with the MiniMax / OpenAI-compatible shims (including mismatched international/domestic base URLs causing 401s, NameErrors, and type mismatches on the new `Result` wrapper) were missed by the test suite due to three primary testing gaps:
+
+### Gap 1: Total Mocking of Initialization Functions
+In the existing tests, the initialization routines (such as `_ensure_minimax_client()` or `_ensure_grok_client()`) were patched out entirely using `unittest.mock.patch`:
+```python
+with patch("src.ai_client._ensure_minimax_client", return_value=MagicMock()):
+ ...
+```
+By replacing the entire initialization function with a dummy mock, the test suite **never executed the actual code** inside those functions. Consequently:
+* The name error when importing modules was bypassed.
+* The lookup of base URLs and API keys in `credentials.toml` was skipped.
+* The parameters passed to `openai.OpenAI()` were never validated.
+
+### Gap 2: Lack of Unit Tests for `_ensure_*_client`
+There were no unit tests focused specifically on client setup/instantiation logic under different credential profiles. The tests only verified what happens *after* a client is assumed to be successfully instantiated.
+
+### Gap 3: Mismatched Mock Return Types
+The test mocks returned raw strings or custom mock responses, rather than validating that the helper senders correctly return a `Result[str]` wrapper (resulting in assertions like `assert result == "hi from grok"` failing because the sender returned `Result(data="hi from grok")`).
+
+---
+
+## 2. Test Hardening Steps Taken
+
+To ensure these regressions are caught automatically in the future, we have implemented the following unit tests in [tests/test_minimax_provider.py](file:///C:/projects/manual_slop/tests/test_minimax_provider.py):
+
+1. **`test_minimax_ensure_client_instantiation`**:
+   * Patches out credentials loading and the warmed `openai` module.
+   * Invokes `_ensure_minimax_client()` directly to force instantiation.
+   * Asserts that `openai.OpenAI()` is called with the exact expected credentials and base URL (e.g. `base_url="https://api.minimax.io/v1"`).
+2. **`test_minimax_ensure_client_missing_key_raises_value_error`**:
+   * Simulates a missing API key in `credentials.toml`.
+   * Asserts that `_ensure_minimax_client()` raises a `ValueError` with the message `"MiniMax API key not found in credentials.toml"`.
+
+Both tests run on a clean global client state and execute the actual implementation code rather than mocking it away.
+
+---
+
+## 3. Future Recommendations for Other Providers
+
+To achieve complete coverage and prevent similar regressions across all other OpenAI-compatible shims (Grok, Llama, Qwen, DeepSeek):
+
+* **Implement Instantiation Tests**: Replicate the `test_minimax_ensure_client_instantiation` pattern for `_ensure_grok_client()`, `_ensure_llama_client()`, `_ensure_qwen_client()`, and `_ensure_deepseek_client()`.
+* **Validate Credentials Handling**: Test each provider's error response when keys are missing or invalid in `credentials.toml`.
+* **Type-Check Return Values**: Ensure all provider integration tests assert that their returns match the `Result[str]` type, accessing the response text via `result.data`.