chore(issue.md): remove outdated issue documentation for LLM Tool Call Fallback Feature as it is no longer relevant

This commit is contained in:
Ariel Frischer 2025-02-14 14:07:51 -08:00
parent 56ddd967c0
commit 9f78e7d36c
1 changed files with 0 additions and 90 deletions

View File

@ -1,90 +0,0 @@
# LLM Tool Call Fallback Feature
## Overview
Add functionality to automatically fallback to alternative LLM models when a tool call experiences multiple consecutive failures.
## Background
Currently, when a tool call fails due to LLM-related errors (e.g., invalid format), there is no automatic fallback mechanism. This often causes infinite loop of erroring tool calls.
## Implementation Details
### Configuration
- Add new configuration value `max_tool_failures` (default: 3) to track consecutive failures before triggering fallback
- Add new command line argument `--no-fallback-tool` to disable fallback behavior (enabled by default)
- **Add new command line argument** `--fallback-tool-models` to specify a comma-separated list of fallback tool models (default: "gpt-3.5-turbo,gpt-4")
This list defines the fallback model sequence used by forced tool calls (via `bind_tools`) when tool call failures occur.
- Track failure count per tool call context
- Reset failure counter on successful tool call
- Store fallback model sequence per provider
- Need to validate if ENV vars are set for provider usage of that fallback model
before usage, if that fallback ENV is not available then fallback to the next model
- Have default list of common models, first try `claude-3-5-sonnet-20241022` but
have many alternative fallback models.
### Tool Call Wrapper
Create a new wrapper function to handle tool call execution with fallback logic:
```python
def execute_tool_with_fallback(tool_call_func, *args, **kwargs):
failures = 0
max_failures = get_config().max_tool_failures
while failures < max_failures:
try:
return tool_call_func(*args, **kwargs)
except LLMError as e:
failures += 1
if failures >= max_failures:
# Use forced tool call via bind_tools with retry:
llm_retry = llm_model.with_retry(stop_after_attempt=3) # Try three times
try_fallback_model(force=True, model=llm_retry)
# Merge fallback model chat messages back into the original chat history.
merge_fallback_chat_history()
failures = 0 # Reset counter for new model
else:
raise
```
The prompt passed to `try_fallback_model`, should be the failed last few failing tool calls.
### Model Fallback Sequence
Define fallback sequences for each provider based on model capabilities:
1. Try same provider's smaller models
2. Try alternative providers' similar models
3. Raise final error if all fallbacks fail
## Risks and Mitigations
1. **Cost**
- Risk: Fallback to more expensive models
- Mitigation: Configure cost limits and preferred fallback sequences
2. **State Management**
- Risk: Loss of context during fallbacks
- Mitigation: Preserve conversation state and tool context
## Relevant Files
- ra_aid/agents/ciayn_agent.py
- ra_aid/llm.py
- ra_aid/agent_utils.py
- ra_aid/__main__.py
- ra_aid/models_params.py
## Acceptance Criteria
1. Tool calls automatically attempt fallback models after N consecutive failures
2. `--no-fallback-tool` argument successfully disables fallback behavior
3. Fallback sequence respects provider and model capabilities
4. Original error is preserved if all fallbacks fail
5. Unit tests cover fallback scenarios and edge cases
6. README.md updated to reflect new behavior
## Documentation Updates
1. Add fallback feature to main README
2. Document `--no-fallback-tool` in CLI help
3. Document provider-specific fallback sequences
## Future Considerations
1. Allow custom fallback sequences via configuration
2. Add monitoring and alerting for fallback frequency
3. Optimize fallback selection based on historical success rates
4. Cost-aware fallback routing