Optimizing LLM Costs with TOON: Practical Token Reduction Strategies

LLM API costs can quickly add up, especially when processing large datasets. TOON format can reduce your costs by 30-60%. Here's how to implement it effectively.

Understanding LLM Pricing

Real-World Cost Savings

Let's calculate savings for a real e-commerce product catalog:

import tiktoken

def calculate_cost(text, model="gpt-4-turbo"):
    enc = tiktoken.encoding_for_model(model)
    tokens = len(enc.encode(text))
    cost_per_token = 10 / 1_000_000  # $10 per 1M for GPT-4 Turbo input
    return tokens, tokens * cost_per_token

# For 1000 products:
# JSON: ~45,000 tokens = $0.45
# TOON: ~18,000 tokens = $0.18
# Savings: 60% = $0.27 per request

Monthly Cost Projection

def project_monthly_savings(daily_requests, records_per_request):
    """Calculate monthly savings from JSON to TOON conversion"""
    
    # Approximate tokens per record
    json_tokens_per_record = 45
    toon_tokens_per_record = 18
    
    # Monthly calculations
    monthly_requests = daily_requests * 30
    total_records = monthly_requests * records_per_request
    
    json_total_tokens = total_records * json_tokens_per_record
    toon_total_tokens = total_records * toon_tokens_per_record
    
    # Cost calculation (GPT-4 Turbo: $10/1M input)
    cost_per_million = 10
    json_cost = (json_total_tokens / 1_000_000) * cost_per_million
    toon_cost = (toon_total_tokens / 1_000_000) * cost_per_million
    
    return {
        "json_cost": json_cost,
        "toon_cost": toon_cost,
        "monthly_savings": json_cost - toon_cost,
        "savings_percent": (1 - toon_cost/json_cost) * 100
    }

# Example: E-commerce app with 100 daily product analysis requests
result = project_monthly_savings(daily_requests=100, records_per_request=500)

print("Monthly JSON cost: $" + str(result['json_cost']))
print("Monthly TOON cost: $" + str(result['toon_cost']))
print("Monthly savings: $" + str(result['monthly_savings']))
# Output:
# Monthly JSON cost: $675.00
# Monthly TOON cost: $270.00
# Monthly savings: $405.00 per month!

Implementation Pattern: Preprocessing Pipeline

class LLMDataPipeline {
  constructor(apiKey) {
    this.apiKey = apiKey;
  }
  
  async query(data, question, options = {}) {
    const useToon = options.optimizeTokens !== false;
    
    // Convert to TOON for token optimization
    const formattedData = useToon 
      ? this.jsonToToon(data)
      : JSON.stringify(data, null, 2);
    
    const prompt = this.buildPrompt(formattedData, question, useToon);
    
    // Track token usage
    const inputTokens = this.countTokens(prompt);
    console.log('Input tokens: ' + inputTokens + ' (' + (useToon ? 'TOON' : 'JSON') + ')');
    
    return await this.callLLM(prompt);
  }
  
  buildPrompt(data, question, isToon) {
    const formatNote = isToon 
      ? 'Data is in TOON format (tabular arrays with field headers)'
      : 'Data is in JSON format';
    
    return formatNote + '\n\nDATA:\n' + data + '\n\nQUESTION: ' + question;
  }
  
  jsonToToon(data) {
    // ... conversion logic
  }
  
  countTokens(text) {
    // Approximate: 1 token = 4 characters
    return Math.ceil(text.length / 4);
  }
}

Best Practices Checklist

1. Use TOON for uniform arrays - Greatest token savings
2. Pre-process data client-side - Convert before API call
3. Batch similar records - Maximize tabular efficiency
4. Include format hints - Tell LLM about TOON format
5. Monitor token usage - Track actual savings
6. Cache frequent conversions - Avoid redundant processing

ROI Calculator

function calculateROI(params) {
  const dailyRequests = params.dailyRequests;
  const avgRecordsPerRequest = params.avgRecordsPerRequest;
  const modelCostPerMillionTokens = params.modelCostPerMillionTokens;
  const jsonTokensPerRecord = 45;
  const toonTokensPerRecord = 18;
  
  const monthlyRequests = dailyRequests * 30;
  const yearlyRequests = dailyRequests * 365;
  
  const jsonMonthlyTokens = monthlyRequests * avgRecordsPerRequest * jsonTokensPerRecord;
  const toonMonthlyTokens = monthlyRequests * avgRecordsPerRequest * toonTokensPerRecord;
  
  const jsonMonthlyCost = (jsonMonthlyTokens / 1000000) * modelCostPerMillionTokens;
  const toonMonthlyCost = (toonMonthlyTokens / 1000000) * modelCostPerMillionTokens;
  
  return {
    monthlySavings: jsonMonthlyCost - toonMonthlyCost,
    yearlySavings: (jsonMonthlyCost - toonMonthlyCost) * 12,
    savingsPercent: ((jsonMonthlyCost - toonMonthlyCost) / jsonMonthlyCost * 100).toFixed(1)
  };
}

// Example calculation
const roi = calculateROI({
  dailyRequests: 500,
  avgRecordsPerRequest: 100,
  modelCostPerMillionTokens: 10  // GPT-4 Turbo
});

console.log('Monthly savings: $' + roi.monthlySavings.toFixed(2));
console.log('Yearly savings: $' + roi.yearlySavings.toFixed(2));
console.log('Savings: ' + roi.savingsPercent + '%');
// Monthly savings: $405.00
// Yearly savings: $4,860.00
// Savings: 60.0%