← Back to Home

Documentation

Learn how to benchmark models, estimate costs, and architect RAG systems with YSelector.

1. The Workflow

YSelector uses a "Prompt-First" approach to benchmarking. Unlike static leaderboards, we test models against your specific use case.

  1. Definition: Describe your project in plain English. Our AI extracts requirements like HIPAA compliance, budget constraints, and expected traffic.
  2. Architecture: Decide if you need a Knowledge Base (RAG). If yes, we simulate vector storage costs.
  3. Test Case: Enter a production-grade prompt.
  4. Selection: We recommend models based on your constraints. You select which ones to race.
  5. Analysis: We run parallel inference and generate a CFO-ready cost report.

2. Understanding Metrics

Base Cost vs. Total Cost

Base Cost includes Input Tokens + Infrastructure (Vector DB).Total Cost is calculated after the test runs, adding the actual Output Tokens generated by the model.

Tiktoken Accuracy

We use the tiktoken library (used by OpenAI) to count tokens with byte-level precision, ensuring our cost projections are mathematically accurate.

3. Supported Providers

We currently support direct integration and benchmarks for:

  • Azure OpenAI (GPT-4.1 Series)
  • Anthropic (Claude 3.5 Sonnet/Haiku)
  • Google Vertex AI (Gemini 2.5)
  • Meta (Llama 3.3 via Azure)

4. Security & Compliance

Your data security is paramount. When you provide API keys, they are encrypted at rest using AES-256-GCM. We do not store your prompt data permanently; it is processed transiently for the duration of the test run.