free-claude-code/e2e/conftest.py
Ali Khokhar 4e99ddaa5c
Some checks are pending
Post merge / Warm dependency cache (Linux) (push) Waiting to run
Post merge / Warm dependency cache (Windows) (push) Waiting to run
Post merge / Warm dependency cache (macOS) (push) Waiting to run
Post merge / Publish to PyPI (push) Waiting to run
minor: select API keys for fallback models (#2058)
## Why

Fallback rows could choose a model and reasoning effort but always used
the provider's first API key. Users need to choose which numbered key
each fallback uses, including different keys for the same model.

## How

Add a Key N selector and persist a validated `api_key_number`,
defaulting existing entries to Key 1. Reject duplicate model/key pairs
and key removals that leave a saved fallback's selection unavailable.

Carry the number through routing and the request's retained runtime
generation. Cache and construct clients per provider/key pair so
concurrent attempts keep their selected credentials.

Share rate and concurrency capacity across a provider's keys, while
recovery belongs to the actual credential. A healthy selected fallback
key receives an attempt after another key exhausts rate-limit retries.
Reordering a key preserves its recovery state, and replacing it uses a
separate state. Keep the existing retry classification, attempt count,
and backoff.

Refresh key choices after provider edits while preserving unsaved model
drafts. Display removed numbers as unavailable and reset the choice to
Key 1 when a row changes provider. Numbers follow the current order in
the provider's settings. Providers with another authentication method
show a disabled default selection.

The Admin editor targets desktop viewports, as requested. Its existing
rows and add row keep the three selections aligned horizontally.

Local CI passes, including 9,049 Python tests and 277 browser tests.
Added regressions capture actual authentication headers after 403 and
exhausted 429 failures across both public APIs and all three custom
endpoint formats, and check shared capacity, client coalescing, retained
generations, credential reorder/replacement, and Admin edits.


<!-- greptile_comment -->

<!-- greptile_summary -->

<h2><a
href="https://app.greptile.com/api/retrigger?id=79105240"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://greptile-static-assets.s3.amazonaws.com/badges/RetriggerDark.svg?v=2"><source
media="(prefers-color-scheme: light)"
srcset="https://greptile-static-assets.s3.amazonaws.com/badges/Retrigger.svg?v=2"><img
alt="Retrigger"
src="https://greptile-static-assets.s3.amazonaws.com/badges/Retrigger.svg?v=2"
align="right"></picture></a>Confidence Score: 5/5</h2>

<!-- greptile-impact-verdict --> No outstanding blocking findings were
identified. The reviewed changes are safe to merge.

<details open><summary><h3>Summary</h3></summary>

Adds a numbered API-key choice to each fallback model and carries that
choice through routing and provider construction.

- Fallback rows can choose a numbered API key.
- Fallback attempts send requests with their selected API keys.
- Provider clients keep separate keys while sharing provider limits.
</details>

<!-- greptile_confidence_score:5 -->

<sub>Reviews (3) · Last reviewed commit: ["Keep credential recovery
separate from
p..."](c7c33ac61b)
· [Reviewed by
Greptile](https://www.greptile.com/?utm_source=greptile_expert&utm_medium=github&utm_campaign=code_reviews&utm_content=alishahryar1/free-claude-code)</sub>

<!-- /greptile_comment -->
2026-10-12 10:09:36 -07:00

401 lines
13 KiB
Python

"""Isolated browser-test composition for the local Admin UI."""
import asyncio
import socket
import sys
import threading
import time
from collections.abc import AsyncIterator, Iterator, Mapping
from pathlib import Path
import pytest
import uvicorn
from playwright.sync_api import Page
from e2e.artifacts import failure_artifacts_recorder as failure_artifacts_recorder
from e2e.code_support import CodeControl
from e2e.server_shutdown import join_server
from free_claude_code.api.app import create_app
from free_claude_code.api.ports import ApiServices
from free_claude_code.application.model_metadata import ProviderModelInfo
from free_claude_code.config import env_migrations, paths
from free_claude_code.config.env_migrations import recognized_env_keys
from free_claude_code.config.loader import (
ManagedConfigStore,
clear_settings_cache,
get_settings,
)
from free_claude_code.config.settings import Settings
from free_claude_code.core.anthropic.models import MessagesRequest
from free_claude_code.core.openai_responses import OpenAIResponsesRequest
from free_claude_code.core.reasoning import DEFAULT_REASONING_POLICY, ReasoningPolicy
from free_claude_code.harnesses import (
claude_desktop_integration,
claude_integration,
claude_onboarding,
codex_integration,
jetbrains_acp_integration,
vscode_chat_integration,
)
from free_claude_code.providers.base import BaseProvider, ProviderConfig
from free_claude_code.providers.credential_validation import (
CredentialCheck,
CredentialStatus,
)
from free_claude_code.providers.runtime import ProviderRuntime
from free_claude_code.runtime.application import ApplicationRuntime
from free_claude_code.runtime.asgi import RuntimeASGIApp
from free_claude_code.runtime.configuration import ConfigurationService
from free_claude_code.runtime.folder_picker import NativeFolderPicker
from free_claude_code.runtime.provider_manager import ProviderRuntimeManager
from tests.web_tools_support import StubWebToolsClient
class _ModelListingProvider(BaseProvider):
def __init__(
self,
model_infos: frozenset[ProviderModelInfo] = frozenset(),
*,
error: Exception | None = None,
) -> None:
super().__init__(
ProviderConfig(
api_key="browser-test",
base_url="https://provider.invalid/v1",
http_read_timeout=1.0,
http_write_timeout=1.0,
http_connect_timeout=1.0,
proxy=None,
log_raw_sse_events=False,
log_api_error_tracebacks=False,
)
)
self._model_infos = model_infos
self._error = error
def preflight_messages(
self,
request: MessagesRequest,
*,
reasoning: ReasoningPolicy = DEFAULT_REASONING_POLICY,
model_info: ProviderModelInfo | None = None,
) -> None:
return None
def preflight_responses(
self,
request: OpenAIResponsesRequest,
*,
reasoning: ReasoningPolicy = DEFAULT_REASONING_POLICY,
) -> None:
return None
async def cleanup(self) -> None:
return None
async def list_model_infos(self) -> frozenset[ProviderModelInfo]:
if self._error is not None:
raise self._error
return self._model_infos
async def stream_messages(
self,
request: MessagesRequest,
input_tokens: int = 0,
*,
request_id: str | None = None,
response_model: str | None = None,
reasoning: ReasoningPolicy = DEFAULT_REASONING_POLICY,
request_headers: Mapping[str, str] | None = None,
model_info: ProviderModelInfo | None = None,
continuation=None,
) -> AsyncIterator[str]:
if False:
yield ""
async def stream_responses(
self,
request: OpenAIResponsesRequest,
input_tokens: int = 0,
*,
request_id: str | None = None,
response_model: str | None = None,
reasoning: ReasoningPolicy = DEFAULT_REASONING_POLICY,
request_headers: Mapping[str, str] | None = None,
model_info: ProviderModelInfo | None = None,
continuation=None,
) -> AsyncIterator[str]:
if False:
yield ""
@pytest.fixture
def code_control(tmp_path):
return CodeControl(tmp_path / "code")
@pytest.fixture
def admin_client_files():
"""Allow integration tests to seed client files before server startup."""
@pytest.fixture
def provider_load_guard(monkeypatch: pytest.MonkeyPatch) -> Iterator[None]:
attempted: list[str] = []
def forbidden(provider_id: str, *_args):
attempted.append(provider_id)
raise AssertionError(f"Browser fixture loaded real provider: {provider_id}")
monkeypatch.setattr(
"free_claude_code.providers.runtime.runtime._load_constructor", forbidden
)
yield
assert attempted == [], f"Browser fixture loaded real providers: {attempted}"
@pytest.fixture
def admin_base_url(
request: pytest.FixtureRequest,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
code_control: CodeControl,
admin_client_files,
provider_load_guard,
) -> Iterator[str]:
"""Serve one fully isolated Admin application on an OS-assigned port."""
config_dir = tmp_path / ".fcc"
monkeypatch.setenv("DSH_HOME", str(tmp_path / ".dsh"))
monkeypatch.delenv("FCC_DSH_DESKTOP_API_KEY", raising=False)
monkeypatch.setattr(
jetbrains_acp_integration,
"config_path",
lambda: tmp_path / ".jetbrains/acp.json",
)
monkeypatch.setattr(
jetbrains_acp_integration,
"registry_path",
lambda: tmp_path / "jetbrains/installed.json",
)
monkeypatch.setattr(
jetbrains_acp_integration, "system_root", lambda: tmp_path / "jetbrains/systems"
)
monkeypatch.setattr(
claude_desktop_integration, "config_root", lambda: tmp_path / "Claude-3p"
)
monkeypatch.setattr(claude_desktop_integration, "check_unmanaged", lambda: None)
monkeypatch.setattr(
claude_desktop_integration,
"legacy_windows_root",
lambda: tmp_path / "LegacyClaude-3p",
)
monkeypatch.setattr(
codex_integration, "config_path", lambda: tmp_path / ".codex" / "config.toml"
)
monkeypatch.setattr(
vscode_chat_integration,
"config_path",
lambda: tmp_path / "vscode/chatLanguageModels.json",
)
monkeypatch.setattr(
claude_onboarding, "state_path", lambda: tmp_path / ".claude.json"
)
monkeypatch.setattr(
claude_integration,
"settings_path",
lambda: tmp_path / "vscode" / "settings.json",
)
monkeypatch.setenv("HOME", str(tmp_path))
monkeypatch.setenv("USERPROFILE", str(tmp_path))
for key in recognized_env_keys():
monkeypatch.delenv(key, raising=False)
monkeypatch.delenv("FCC_ENV_FILE", raising=False)
monkeypatch.setenv("MODEL", "open_router/e2e-default")
monkeypatch.setenv("OPENROUTER_API_KEY", "e2e-openrouter-key")
monkeypatch.setenv("GROQ_API_KEY", "e2e-groq-key")
monkeypatch.setenv("CLOUDFLARE_API_TOKEN", "e2e-cloudflare-token")
monkeypatch.setenv("MESSAGING_PLATFORM", "none")
monkeypatch.setenv("VOICE_NOTE_ENABLED", "false")
monkeypatch.setenv("FCC_OPEN_BROWSER", "false")
monkeypatch.setenv("PROXY_AUTH_ENABLED", "false")
for key, value in getattr(request, "param", {}).items():
monkeypatch.setenv(key, value)
monkeypatch.setattr(paths, "config_dir_path", lambda: config_dir)
monkeypatch.setattr(env_migrations, "legacy_env_paths", lambda: ())
monkeypatch.setattr(env_migrations, "verified_checkout_env_path", lambda: None)
clear_settings_cache()
async def verified_credentials(settings, submissions):
return tuple(
CredentialCheck(
submission.key,
CredentialStatus.VERIFIED,
"API key accepted.",
index,
submission.provider_id,
)
for submission in submissions
for index, _ in enumerate(submission.credentials)
)
monkeypatch.setattr(
"free_claude_code.runtime.application.check_credentials", verified_credentials
)
store = ManagedConfigStore()
store.initialize({})
store.commit(
dict(store.read({}).managed) | {"ANTHROPIC_AUTH_TOKEN": "e2e-proxy-token"}
)
provider_secret = "CREDENTIAL[unrecognized-format-987654321]"
providers: dict[str, BaseProvider] = {
"nvidia_nim": _ModelListingProvider(),
"open_router": _ModelListingProvider(
frozenset(
{
ProviderModelInfo("vendor/model-a"),
ProviderModelInfo(
"vendor/model-b",
supports_thinking=True,
context_window_tokens=100_000,
max_output_tokens=20_000,
),
ProviderModelInfo(
"vendor/small-context",
supports_thinking=True,
context_window_tokens=8_192,
max_output_tokens=4_096,
),
}
)
),
"groq": _ModelListingProvider(
error=RuntimeError(f"Provider rejected credential {provider_secret}")
),
}
async def fixture_provider(
provider_id: str, _settings: Settings, _admission_registry, _api_key_number
) -> BaseProvider:
if provider_id not in providers:
raise AssertionError(f"Missing browser fixture provider: {provider_id}")
return providers[provider_id]
manager = ProviderRuntimeManager(
get_settings(),
runtime_factory=lambda snapshot, admission_registry: ProviderRuntime(
snapshot,
admission_registry,
{(provider_id, 1): provider for provider_id, provider in providers.items()},
provider_constructor=fixture_provider,
),
)
runtime = ApplicationRuntime(
manager,
configuration=ConfigurationService(ManagedConfigStore()),
transcriber=None,
code_service=code_control.service,
database=code_control.database,
)
monkeypatch.setattr(
NativeFolderPicker,
"_select",
lambda _picker, initial, stop: code_control.folder_picker.select(initial, stop),
)
app = RuntimeASGIApp(
create_app(
ApiServices(
requests=manager,
admin=runtime,
tasks=runtime,
web_tools=StubWebToolsClient(),
code=code_control.service,
)
),
runtime,
)
async def local_provider_result(
provider_id: str,
base_url: str,
path: str,
) -> dict[str, object]:
return {
"provider_id": provider_id,
"status": "reachable",
"label": "Reachable",
"base_url": base_url,
}
monkeypatch.setattr(
"free_claude_code.api.admin_routes._check_local_provider",
local_provider_result,
)
listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listener.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
listener.bind(("127.0.0.1", 0))
listener.listen(128)
port = listener.getsockname()[1]
server = uvicorn.Server(
uvicorn.Config(
app,
log_level="error",
access_log=False,
lifespan="on",
)
)
async def serve() -> None:
code_control.loop = asyncio.get_running_loop()
await server.serve(sockets=[listener])
def run_server() -> None:
if sys.platform == "win32":
# This HTTP-only fixture owns no subprocess transports. A browser
# reset can interrupt Proactor socket cleanup on Python 3.14.0;
# use the selector loop here, leaving Playwright's loop unchanged.
asyncio.run(
serve(),
loop_factory=asyncio.SelectorEventLoop,
)
else:
asyncio.run(serve())
thread = threading.Thread(
target=run_server,
name="fcc-admin-playwright",
daemon=True,
)
thread.start()
deadline = time.monotonic() + 5.0
try:
while not server.started:
if not thread.is_alive():
raise RuntimeError("Admin browser-test server exited during startup")
if time.monotonic() >= deadline:
raise TimeoutError("Admin browser-test server did not start")
time.sleep(0.01)
yield f"http://127.0.0.1:{port}"
finally:
server.should_exit = True
join_server(thread, code_control, request)
listener.close()
clear_settings_cache()
@pytest.fixture(autouse=True)
def close_browser_connections_before_server_teardown(
admin_base_url: str,
page: Page,
) -> Iterator[None]:
del admin_base_url
yield
if not page.is_closed():
page.goto("about:blank")
# Context disposal can otherwise leak an unfinished route.fetch() error
# into the next test's Playwright call.
page.unroute_all(behavior="ignoreErrors")