Model Integration

Comprehensive guide to working with different AI model providers programmatically. Learn how to configure, authenticate, and optimize for each provider's specific capabilities and limitations.

Provider Overview

Umwelten supports multiple AI model providers, each with different characteristics:

Provider	Type	Cost	Best For	Authentication
Google	API	Pay-per-token	General use, vision	API Key
Ollama	Local	Free	Development, privacy	Local server
OpenRouter	API	Pay-per-token	Premium models	API Key
LM Studio	Local	Free	Custom models	Local server
LlamaBarn	Local	Free	llama.cpp on macOS	Local server
llama-swap	Local	Free	Upstream llama.cpp, on-demand model loading	Local server
GitHub Models	API	Free tier	Azure-hosted models	GitHub Token
Fireworks	API	Pay-per-token	Hosted OSS and partner models	API Key
MiniMax	API	Pay-per-token	MiniMax M2 family	API Key

Google Gemini Models

Google provides the most comprehensive model selection with excellent vision capabilities and competitive pricing.

Authentication

typescript

// Set environment variable
process.env.GOOGLE_GENERATIVE_AI_API_KEY = "your-api-key-here";

// Or use .env file
// GOOGLE_GENERATIVE_AI_API_KEY=your-api-key-here

Available Models

typescript

import { ModelDetails } from './core/cognition/types.js';

// Fast and cost-effective
const flashModel: ModelDetails = {
  name: 'gemini-3-flash-preview',
  provider: 'google',
  temperature: 0.7
};

// Highest quality reasoning
const proModel: ModelDetails = {
  name: 'gemini-2.5-pro-exp-03-25',
  provider: 'google',
  temperature: 0.3 // Lower temperature for reasoning tasks
};

// Ultra-fast for simple tasks
const flash8bModel: ModelDetails = {
  name: 'gemini-1.5-flash-8b',
  provider: 'google',
  temperature: 1.0
};

Vision Capabilities

Google models excel at image and document analysis:

typescript

import { Interaction } from './core/interaction/core/interaction.js';
import { Stimulus } from './core/stimulus/stimulus.js';
import { BaseModelRunner } from './core/cognition/runner.js';

// Image analysis
const visionStimulus = new Stimulus({ role: "expert image analyst" });
const visionAnalysis = new Interaction(
  { name: 'gemini-3-flash-preview', provider: 'google' },
  visionStimulus
);

await visionAnalysis.addAttachmentFromPath('./image.jpg');
visionAnalysis.addMessage({
  role: 'user',
  content: 'Describe this image in detail, including objects, colors, and composition.'
});

const runner = new BaseModelRunner();
const response = await runner.generateText(visionAnalysis);

Long Context Handling

Google models support very long contexts (up to 2M tokens):

typescript

// Process large documents
const docStimulus = new Stimulus({ role: "document analyst", objective: "analyze documents comprehensively" });
const longContext = new Interaction(
  { name: 'gemini-3-flash-preview', provider: 'google' },
  docStimulus
);

await longContext.addAttachmentFromPath('./large-report.pdf');
longContext.addMessage({
  role: 'user',
  content: 'Extract key themes, recommendations, and action items from this document.'
});

const response = await runner.generateText(longContext);

Cost Optimization

typescript

// Use Flash for cost-sensitive applications
const costEffective: ModelDetails = {
  name: 'gemini-3-flash-preview',
  provider: 'google',
  maxTokens: 500,  // Limit output length to control costs
  temperature: 0.1 // Lower temperature for consistent, focused responses
};

// Use Pro only for complex reasoning
const premiumReasoning: ModelDetails = {
  name: 'gemini-2.5-pro-exp-03-25',
  provider: 'google',
  temperature: 0.2
};

Ollama Local Models

Ollama provides free local model execution with privacy benefits but requires local setup.

Setup

bash

# Install Ollama (one-time setup)
curl -fsSL https://ollama.ai/install.sh | sh

# Start Ollama server
ollama serve

# Pull models
ollama pull gemma3:12b
ollama pull codestral:latest
ollama pull qwen2.5vl:latest  # Vision model

Configuration

typescript

// Default local configuration
const ollamaModel: ModelDetails = {
  name: 'gemma3:12b',
  provider: 'ollama'
  // Uses default OLLAMA_HOST=http://localhost:11434
};

// Custom Ollama server
process.env.OLLAMA_HOST = "http://custom-server:11434";
const customOllama: ModelDetails = {
  name: 'gemma3:27b',
  provider: 'ollama'
};

Available Models

typescript

// General purpose models
const gemma12b: ModelDetails = {
  name: 'gemma3:12b',
  provider: 'ollama',
  temperature: 0.8
};

const gemma27b: ModelDetails = {
  name: 'gemma3:27b',
  provider: 'ollama',
  temperature: 0.7 // Larger model, can use slightly lower temperature
};

// Code-specialized models
const codestral: ModelDetails = {
  name: 'codestral:latest',
  provider: 'ollama',
  temperature: 0.1 // Low temperature for precise code generation
};

// Vision models
const qwenVision: ModelDetails = {
  name: 'qwen2.5vl:latest',
  provider: 'ollama',
  temperature: 0.5
};

Local Model Advantages

typescript

// No API costs
async function freeAnalysis(texts: string[]): Promise<void> {
  const model: ModelDetails = {
    name: 'gemma3:12b',
    provider: 'ollama'
  };
  
  const runner = new BaseModelRunner();
  
  // Process unlimited texts without cost concerns
  for (const text of texts) {
    const stimulus = new Stimulus({ role: "text analyst" });
    const conversation = new Interaction(model, stimulus);
    conversation.addMessage({ role: 'user', content: text });
    
    const response = await runner.generateText(conversation);
    console.log(`Analysis: ${response.content}`);
  }
}

// Privacy-focused processing
async function privateDocumentAnalysis(documentPath: string): Promise<void> {
  const model: ModelDetails = {
    name: 'gemma3:27b',
    provider: 'ollama'
  };
  
  const stimulus = new Stimulus({ role: "document analyst" });
  const conversation = new Interaction(model, stimulus);
  await conversation.addAttachmentFromPath(documentPath); // Never leaves your machine
  
  const runner = new BaseModelRunner();
  const response = await runner.generateText(conversation);
  // Document content and analysis stay completely local
}

Performance Considerations

typescript

// Optimize for local hardware
const lightweightModel: ModelDetails = {
  name: 'gemma3:12b',    // Smaller model for faster responses
  provider: 'ollama',
  maxTokens: 500,        // Limit output for speed
  temperature: 0.5
};

// High-quality model for important tasks
const qualityModel: ModelDetails = {
  name: 'gemma3:27b',    // Larger model for better quality
  provider: 'ollama',
  maxTokens: 2000,
  temperature: 0.7
};

OpenRouter Hosted Models

OpenRouter provides access to the latest models from multiple providers through a single API.

Authentication

typescript

// Set environment variable
process.env.OPENROUTER_API_KEY = "your-api-key-here";

Premium Models

typescript

// GPT-4o (highest quality)
const gpt4o: ModelDetails = {
  name: 'openai/gpt-4o',
  provider: 'openrouter',
  temperature: 0.7,
  maxTokens: 2000
};

// GPT-4o Mini (balanced cost/quality)
const gpt4oMini: ModelDetails = {
  name: 'openai/gpt-4o-mini',
  provider: 'openrouter',
  temperature: 0.8,
  maxTokens: 1500
};

// Claude Sonnet (excellent for analysis)
const claudeSonnet: ModelDetails = {
  name: 'anthropic/claude-3.7-sonnet:thinking',
  provider: 'openrouter',
  temperature: 0.3, // Lower temperature for analytical tasks
  maxTokens: 3000
};

Model Selection Strategy

typescript

// Choose models based on task complexity
async function smartModelSelection(task: string, complexity: 'simple' | 'medium' | 'complex') {
  let model: ModelDetails;
  
  switch (complexity) {
    case 'simple':
      model = {
        name: 'openai/gpt-4o-mini',
        provider: 'openrouter',
        temperature: 0.5
      };
      break;
      
    case 'medium':
      model = {
        name: 'anthropic/claude-3.7-sonnet:thinking',
        provider: 'openrouter',
        temperature: 0.3
      };
      break;
      
    case 'complex':
      model = {
        name: 'openai/gpt-4o',
        provider: 'openrouter',
        temperature: 0.2
      };
      break;
  }
  
  const stimulus = new Stimulus({ role: "expert analyst" });
  const conversation = new Interaction(model, stimulus);
  conversation.addMessage({ role: 'user', content: task });
  
  const runner = new BaseModelRunner();
  return runner.generateText(conversation);
}

Cost Management

typescript

// Track costs across OpenRouter models
async function costAwareProcessing(tasks: string[]): Promise<void> {
  let totalCost = 0;
  const maxBudget = 5.00; // $5 budget
  
  const model: ModelDetails = {
    name: 'openai/gpt-4o-mini',
    provider: 'openrouter'
  };
  
  const runner = new BaseModelRunner();
  
  for (const task of tasks) {
    if (totalCost >= maxBudget) {
      console.log(`Budget limit reached: $${totalCost}`);
      break;
    }
    
    const stimulus = new Stimulus({ role: "task processor" });
    const conversation = new Interaction(model, stimulus);
    conversation.addMessage({ role: 'user', content: task });
    
    const response = await runner.generateText(conversation);
    
    if (response.metadata?.cost) {
      totalCost += response.metadata.cost.totalCost;
      console.log(`Task cost: $${response.metadata.cost.totalCost}, Total: $${totalCost}`);
    }
  }
}

LM Studio Local Models

LM Studio provides a user-friendly way to run local models with a GUI interface.

Setup

Download and install LM Studio
Load a model through the GUI
Start the local server
Configure Umwelten to use the server

Configuration

typescript

// Default LM Studio configuration
const lmStudioModel: ModelDetails = {
  name: 'local-model-name', // Use the model name as shown in LM Studio
  provider: 'lmstudio'
  // Uses default LMSTUDIO_BASE_URL=http://localhost:1234
};

// Custom LM Studio server
process.env.LMSTUDIO_BASE_URL = "http://custom-host:1234";
const customLMStudio: ModelDetails = {
  name: 'custom-model',
  provider: 'lmstudio'
};

Model Management

typescript

// Check available models in LM Studio
async function listLMStudioModels(): Promise<void> {
  try {
    const response = await fetch('http://localhost:1234/v1/models');
    const models = await response.json();
    
    console.log('Available LM Studio models:');
    models.data.forEach(model => {
      console.log(`- ${model.id}`);
    });
  } catch (error) {
    console.error('LM Studio server not running or not accessible');
  }
}

Specialized Model Usage

typescript

// Use code-specialized models in LM Studio
const codeModel: ModelDetails = {
  name: 'CodeLlama-13B-Instruct', // Example model name in LM Studio
  provider: 'lmstudio',
  temperature: 0.1,
  maxTokens: 2000
};

async function codeGeneration(prompt: string): Promise<string> {
  const stimulus = new Stimulus({ role: "expert programmer" });
  const conversation = new Interaction(codeModel, stimulus);
  conversation.addMessage({ role: 'user', content: prompt });
  
  const runner = new BaseModelRunner();
  const response = await runner.generateText(conversation);
  
  return response.content;
}

LlamaBarn Local Models

LlamaBarn is a 12 MB macOS menu bar app that runs local GGUF models on llama.cpp with an OpenAI-compatible server. Models sleep idle and wake on demand.

Setup

Install LlamaBarn (menu bar app)
Add a model from the menu bar (downloads from Hugging Face)
The server runs on http://localhost:2276/v1 by default

Configuration

typescript

const llamaBarnModel: ModelDetails = {
  name: 'glm-4.7-flash', // Or any id from listModels()
  provider: 'llamabarn'
};

// Override host if needed
process.env.LLAMABARN_HOST = 'http://localhost:2276/v1';

Model Management

typescript

import { createLlamaBarnProvider } from './core/providers/llamabarn.js';

const provider = createLlamaBarnProvider();
const models = await provider.listModels();

// Each model reports loaded / sleeping / unloaded state
for (const m of models) {
  console.log(`${m.name} [${m.details?.state}] ctx=${m.contextLength}`);
}

Sleeping models wake on the first request — expect a cold-start pause (often 60s+) before the first token. The contextLength is parsed from LlamaBarn's per-model preset (ctx-size).

llama-swap Local Models

llama-swap is a small proxy that launches upstream llama-server on demand per model alias and unloads idle models after a configurable TTL. It pairs well with the brew install llama.cpp toolchain and gives you "Ollama-style hot-swapping with real upstream llama.cpp."

Setup

Install the binaries: brew install llama.cpp and brew install llama-swap (or go install github.com/mostlygeek/llama-swap@latest)

Generate a config from your local GGUF caches:

bash

pnpm run cli -- models llamaswap-config \
  --output llama-swap.yaml \
  --llama-server-path /opt/homebrew/bin/llama-server

Start the proxy:

bash

llama-swap --config llama-swap.yaml --listen :8080

Configuration

typescript

const llamaSwapModel: ModelDetails = {
  name: 'gemma-4-26b-a4b', // alias from llama-swap config
  provider: 'llamaswap'
};

// Override host if running llama-swap on a different port
process.env.LLAMASWAP_HOST = 'http://localhost:8090/v1';

Config Generation (programmatic)

typescript

import {
  generateLlamaSwapConfig,
  findLlamaSwapModels,
  buildLlamaSwapConfig,
} from './core/providers/llamaswap-config.js';

// One-shot: scan default paths + emit YAML
const { yaml, models } = generateLlamaSwapConfig({
  ttlSeconds: 300,
  ctxSize: 0,
  extraCachePaths: ['/my/extra/gguf/dir'],
});

// Or compose the two halves:
const gguf = findLlamaSwapModels({ extraCachePaths: ['/my/extra/gguf/dir'] });
const yaml2 = buildLlamaSwapConfig(gguf, {
  ttlSeconds: 60,
  llamaServerPath: '/opt/homebrew/bin/llama-server',
  extraArgs: ['--flash-attn'],
});

The scanner follows symlinks (the Hugging Face hub cache stores snapshots as symlinks to blob files), dedupes multi-quant copies by preferring the largest file, and skips non-chat GGUFs (mmproj projector sidecars, embedding models).

Cross-Provider Strategies

Model Cascading

Use cheaper models for initial screening, expensive models for detailed analysis:

typescript

async function cascadedAnalysis(content: string): Promise<{ screening: string, detailed: string }> {
  // First pass: Quick screening with cost-effective model
  const screeningModel: ModelDetails = {
    name: 'gemma3:12b',
    provider: 'ollama'
  };
  
  const screeningStimulus = new Stimulus({ role: "content screener" });
  const screening = new Interaction(screeningModel, screeningStimulus);
  screening.addMessage({
    role: 'user',
    content: `Is this content worth detailed analysis? Respond briefly: ${content}`
  });
  
  const runner = new BaseModelRunner();
  const screeningResult = await runner.generateText(screening);
  
  // Second pass: Detailed analysis only if warranted
  let detailedResult = { content: 'Analysis not needed' };
  
  if (screeningResult.content.toLowerCase().includes('yes') || 
      screeningResult.content.toLowerCase().includes('worth')) {
    
    const detailedModel: ModelDetails = {
      name: 'openai/gpt-4o',
      provider: 'openrouter'
    };
    
    const detailedStimulus = new Stimulus({ role: "content analyst", objective: "comprehensive analysis" });
    const detailed = new Interaction(detailedModel, detailedStimulus);
    detailed.addMessage({
      role: 'user',
      content: `Provide detailed analysis: ${content}`
    });
    
    detailedResult = await runner.generateText(detailed);
  }
  
  return {
    screening: screeningResult.content,
    detailed: detailedResult.content
  };
}

Fallback Strategies

Implement robust fallback between providers:

typescript

async function robustGeneration(prompt: string): Promise<ModelResponse> {
  const fallbackChain: ModelDetails[] = [
    { name: 'gemini-3-flash-preview', provider: 'google' },      // Try Google first
    { name: 'openai/gpt-4o-mini', provider: 'openrouter' }, // Fallback to OpenRouter
    { name: 'gemma3:12b', provider: 'ollama' }              // Final fallback to local
  ];
  
  const runner = new BaseModelRunner();
  
  for (const model of fallbackChain) {
    try {
      const stimulus = new Stimulus({ role: "assistant" });
      const conversation = new Interaction(model, stimulus);
      conversation.addMessage({ role: 'user', content: prompt });
      
      const response = await runner.generateText(conversation);
      console.log(`Success with ${model.provider}:${model.name}`);
      return response;
      
    } catch (error) {
      console.warn(`Failed with ${model.provider}:${model.name}:`, error.message);
      // Continue to next model in chain
    }
  }
  
  throw new Error('All providers failed');
}

Performance Comparison

typescript

async function benchmarkProviders(prompt: string): Promise<void> {
  const models: ModelDetails[] = [
    { name: 'gemini-3-flash-preview', provider: 'google' },
    { name: 'gemma3:12b', provider: 'ollama' },
    { name: 'openai/gpt-4o-mini', provider: 'openrouter' }
  ];
  
  const results = [];
  const runner = new BaseModelRunner();
  
  for (const model of models) {
    const startTime = Date.now();
    
    try {
      const stimulus = new Stimulus({ role: "assistant" });
      const conversation = new Interaction(model, stimulus);
      conversation.addMessage({ role: 'user', content: prompt });
      
      const response = await runner.generateText(conversation);
      const duration = Date.now() - startTime;
      
      results.push({
        provider: `${model.provider}:${model.name}`,
        duration,
        tokens: response.metadata?.tokenUsage?.total || 0,
        cost: response.metadata?.cost?.totalCost || 0,
        content: response.content.substring(0, 100) + '...'
      });
      
    } catch (error) {
      results.push({
        provider: `${model.provider}:${model.name}`,
        error: error.message
      });
    }
  }
  
  // Display performance comparison
  console.table(results);
}

Provider-Specific Best Practices

Google Optimization

typescript

// Optimize for Google's strengths
const googleBestPractices: ModelDetails = {
  name: 'gemini-3-flash-preview',
  provider: 'google',
  temperature: 0.7,
  // Google handles long contexts well - don't artificially limit
  // Google excels at vision tasks - use for image/document analysis
  // Google pricing is token-based - optimize prompt length
};

Ollama Optimization

typescript

// Optimize for local resources
const ollamaBestPractices: ModelDetails = {
  name: 'gemma3:12b',
  provider: 'ollama',
  temperature: 0.8,
  maxTokens: 1000, // Limit for faster local generation
  // Use for privacy-sensitive data
  // Great for development and testing
  // Consider hardware limitations
};

OpenRouter Optimization

typescript

// Optimize for premium model access
const openRouterBestPractices: ModelDetails = {
  name: 'openai/gpt-4o-mini',
  provider: 'openrouter',
  temperature: 0.6,
  // Monitor costs carefully
  // Use for complex reasoning tasks
  // Leverage latest model access
};

Error Handling by Provider

Provider-Specific Error Patterns

typescript

async function handleProviderErrors(model: ModelDetails, conversation: Interaction): Promise<ModelResponse | null> {
  const runner = new BaseModelRunner();
  
  try {
    return await runner.generateText(conversation);
  } catch (error) {
    switch (model.provider) {
      case 'google':
        if (error.message.includes('quota exceeded')) {
          console.error('Google API quota exceeded - try again later');
        } else if (error.message.includes('safety')) {
          console.error('Content triggered safety filters - adjust prompt');
        }
        break;
        
      case 'ollama':
        if (error.message.includes('connection refused')) {
          console.error('Ollama server not running - start with `ollama serve`');
        } else if (error.message.includes('model not found')) {
          console.error('Model not pulled - run `ollama pull model-name`');
        }
        break;
        
      case 'openrouter':
        if (error.message.includes('insufficient credits')) {
          console.error('OpenRouter credits exhausted - add more credits');
        } else if (error.message.includes('model not found')) {
          console.error('Model not available on OpenRouter - check model list');
        }
        break;
        
      case 'lmstudio':
        if (error.message.includes('connection refused')) {
          console.error('LM Studio server not running - start server in LM Studio');
        }
        break;
    }
    
    return null;
  }
}

Next Steps

Explore Evaluation Framework for advanced multi-provider evaluation patterns
See Schema Validation for structured output across different providers
Check Core Classes for detailed API documentation

Model Integration ​

Provider Overview ​

Google Gemini Models ​

Authentication ​

Available Models ​

Vision Capabilities ​

Long Context Handling ​

Cost Optimization ​

Ollama Local Models ​

Setup ​

Configuration ​

Available Models ​

Local Model Advantages ​

Performance Considerations ​

OpenRouter Hosted Models ​

Authentication ​

Premium Models ​

Model Selection Strategy ​

Cost Management ​

LM Studio Local Models ​

Setup ​

Configuration ​

Model Management ​

Specialized Model Usage ​

LlamaBarn Local Models ​

Setup ​

Configuration ​

Model Management ​

llama-swap Local Models ​

Setup ​

Configuration ​

Config Generation (programmatic) ​

Cross-Provider Strategies ​

Model Cascading ​

Fallback Strategies ​

Performance Comparison ​

Provider-Specific Best Practices ​

Google Optimization ​

Ollama Optimization ​

OpenRouter Optimization ​

Error Handling by Provider ​

Provider-Specific Error Patterns ​

Next Steps ​

Model Integration

Provider Overview

Google Gemini Models

Authentication

Available Models

Vision Capabilities

Long Context Handling

Cost Optimization

Ollama Local Models

Setup

Configuration

Available Models

Local Model Advantages

Performance Considerations

OpenRouter Hosted Models

Authentication

Premium Models

Model Selection Strategy

Cost Management

LM Studio Local Models

Setup

Configuration

Model Management

Specialized Model Usage

LlamaBarn Local Models

Setup

Configuration

Model Management

llama-swap Local Models

Setup

Configuration

Config Generation (programmatic)

Cross-Provider Strategies

Model Cascading

Fallback Strategies

Performance Comparison

Provider-Specific Best Practices

Google Optimization

Ollama Optimization

OpenRouter Optimization

Error Handling by Provider

Provider-Specific Error Patterns

Next Steps