Execution API

Query and analyze execution history, metrics, costs, and performance data for prompts and chains.

Overview

The Execution API provides access to detailed execution history for both prompts and chains. Every execution is recorded with comprehensive metrics including latency, token usage, costs, and status information.

All API requests require authentication via API key in the X-API-Key header. See the Authentication guide for details.

Queries

Get All Executions

query

executions

Retrieve all prompt executions for the authenticated user

query GetExecutions(
  $limit: Int
  $offset: Int
  $status: ExecutionStatus
) {
  executions(
    limit: $limit
    offset: $offset
    status: $status
  ) {
    id
    status
    output
    error
    latencyMs
    tokenIn
    tokenOut
    costUsd
    createdAt
    promptVersion {
      version
      prompt {
        id
        name
      }
    }
    user {
      id
      email
    }
  }
}

Parameter	Type	Description
`limit`	`number`	Maximum number of executions to return (default: 50)
`offset`	`number`	Number of executions to skip for pagination
`status`	`ExecutionStatus`	Filter by status (success, error, pending)

Example Response

json

{
  "data": {
    "executions": [
      {
        "id": "exec_abc123",
        "status": "success",
        "output": "- Sentiment: Positive\n- Confidence: 0.95",
        "error": null,
        "latencyMs": 1250,
        "tokenIn": 45,
        "tokenOut": 28,
        "costUsd": "0.0012",
        "createdAt": "2025-01-21T10:30:00Z",
        "promptVersion": {
          "version": 1,
          "prompt": {
            "id": "cm1234567890",
            "name": "sentiment-analyzer"
          }
        },
        "user": {
          "id": "user_xyz789",
          "email": "user@example.com"
        }
      }
    ]
  }
}

Get Single Execution

query

execution

Retrieve detailed information about a specific execution

query GetExecution($id: ID!) {
  execution(id: $id) {
    id
    status
    input
    output
    error
    latencyMs
    tokenIn
    tokenOut
    costUsd
    createdAt
    promptVersion {
      id
      version
      content
      params
      prompt {
        id
        name
        description
      }
    }
  }
}

Parameter	Type	Description
`id` required	`ID!`	Unique identifier for the execution

Get Chain Executions

query

chainExecutions

Retrieve all chain executions with step-by-step details

query GetChainExecutions(
  $limit: Int
  $offset: Int
  $chainId: ID
) {
  chainExecutions(
    limit: $limit
    offset: $offset
    chainId: $chainId
  ) {
    id
    status
    input
    output
    error
    totalLatency
    totalCost
    stepCount
    createdAt
    chain {
      id
      name
      description
    }
    steps {
      order
      type
      output
      latency
      cost
      status
      error
    }
  }
}

Parameter	Type	Description
`limit`	`number`	Maximum number of executions to return (default: 50)
`offset`	`number`	Number of executions to skip for pagination
`chainId`	`ID`	Filter executions for a specific chain

Example Response

json

{
  "data": {
    "chainExecutions": [
      {
        "id": "exec_chain_abc123",
        "status": "success",
        "input": {
          "message": "I need help with my order"
        },
        "output": "I'd be happy to help you with your order. Could you please provide your order number?",
        "error": null,
        "totalLatency": 2650,
        "totalCost": 0.0033,
        "stepCount": 2,
        "createdAt": "2025-01-21T10:35:00Z",
        "chain": {
          "id": "cm9876543210",
          "name": "customer-support-analyzer",
          "description": "Analyzes customer support messages"
        },
        "steps": [
          {
            "order": 0,
            "type": "PROMPT",
            "output": "{\"sentiment\": \"Neutral\", \"confidence\": 0.75}",
            "latency": 1200,
            "cost": 0.0015,
            "status": "success",
            "error": null
          },
          {
            "order": 1,
            "type": "PROMPT",
            "output": "I'd be happy to help you with your order...",
            "latency": 1450,
            "cost": 0.0018,
            "status": "success",
            "error": null
          }
        ]
      }
    ]
  }
}

Get Execution Statistics

query

executionStats

Retrieve aggregated statistics for executions

query GetExecutionStats(
  $startDate: DateTime
  $endDate: DateTime
) {
  executionStats(
    startDate: $startDate
    endDate: $endDate
  ) {
    totalExecutions
    successfulExecutions
    failedExecutions
    successRate
    averageLatency
    totalTokensIn
    totalTokensOut
    totalCost
    executionsByDay {
      date
      count
      successCount
      errorCount
    }
    topPrompts {
      promptId
      promptName
      executionCount
      averageCost
      averageLatency
    }
  }
}

Parameter	Type	Description
`startDate`	`DateTime`	Start date for statistics (ISO 8601 format)
`endDate`	`DateTime`	End date for statistics (ISO 8601 format)

Example Response

json

{
  "data": {
    "executionStats": {
      "totalExecutions": 1250,
      "successfulExecutions": 1187,
      "failedExecutions": 63,
      "successRate": 94.96,
      "averageLatency": 1340,
      "totalTokensIn": 45000,
      "totalTokensOut": 32000,
      "totalCost": 1.45,
      "executionsByDay": [
        {
          "date": "2025-01-20",
          "count": 150,
          "successCount": 142,
          "errorCount": 8
        },
        {
          "date": "2025-01-21",
          "count": 175,
          "successCount": 168,
          "errorCount": 7
        }
      ],
      "topPrompts": [
        {
          "promptId": "cm1234567890",
          "promptName": "sentiment-analyzer",
          "executionCount": 450,
          "averageCost": 0.0012,
          "averageLatency": 1250
        }
      ]
    }
  }
}

Execution Status

Parameter	Type	Description
`pending`	`ExecutionStatus`	Execution has been initiated but not yet completed
`success`	`ExecutionStatus`	Execution completed successfully with output available
`error`	`ExecutionStatus`	Execution failed with error message available

Metrics Explained

Performance Metrics

Parameter	Type	Description
`latencyMs`	`number`	Total execution time in milliseconds (including API calls, model inference, network overhead)
`tokenIn`	`number`	Number of tokens in the input (prompt template + variable substitution)
`tokenOut`	`number`	Number of tokens generated in the output response
`costUsd`	`string`	Exact cost in USD with 4 decimal precision

Cost Calculation

Costs are calculated based on the AI model used and token consumption:

Claude 3.5 Sonnet Pricing (as of Jan 2025):
Input:  $3.00 / 1M tokens
Output: $15.00 / 1M tokens

Example calculation:
Input tokens:  500 × $3.00 / 1,000,000 = $0.0015
Output tokens: 200 × $15.00 / 1,000,000 = $0.0030
Total cost: $0.0045

Costs vary by model. Claude Opus and GPT-4 are typically more expensive than Haiku or GPT-3.5 Turbo. Check current pricing for your selected models.

Error Handling

Failed Execution

{
  "data": {
    "execution": {
      "id": "exec_abc123",
      "status": "error",
      "output": null,
      "error": "Invalid API key for provider 'anthropic'",
      "latencyMs": 150,
      "tokenIn": 0,
      "tokenOut": 0,
      "costUsd": "0.0000",
      "createdAt": "2025-01-21T10:30:00Z"
    }
  }
}

Common Error Types

Parameter	Type	Description
`Authentication Error`	`Error`	Invalid API keys, expired credentials, or insufficient permissions
`Validation Error`	`Error`	Missing required variables, invalid input types, or schema mismatches
`Rate Limit Error`	`Error`	Too many requests to the AI provider within a time window
`Timeout Error`	`Error`	Execution exceeded maximum allowed time
`Model Error`	`Error`	AI model returned an error or unexpected response

Code Examples

JavaScript / TypeScript

Fetch Execution Statistics

typescript

const apiKey = 'your-api-key'
const endpoint = 'https://api.promptforge.sh/graphql'

async function getExecutionStats(days: number = 7) {
  const endDate = new Date()
  const startDate = new Date()
  startDate.setDate(startDate.getDate() - days)

  const response = await fetch(endpoint, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-API-Key': apiKey,
    },
    body: JSON.stringify({
      query: `
        query GetExecutionStats($startDate: DateTime, $endDate: DateTime) {
          executionStats(startDate: $startDate, endDate: $endDate) {
            totalExecutions
            successfulExecutions
            failedExecutions
            successRate
            averageLatency
            totalCost
            topPrompts {
              promptName
              executionCount
              averageCost
            }
          }
        }
      `,
      variables: {
        startDate: startDate.toISOString(),
        endDate: endDate.toISOString(),
      },
    }),
  })

  const { data, errors } = await response.json()

  if (errors) {
    throw new Error(errors[0].message)
  }

  return data.executionStats
}

// Usage
const stats = await getExecutionStats(30)

console.log(`Total executions: ${stats.totalExecutions}`)
console.log(`Success rate: ${stats.successRate}%`)
console.log(`Total cost: $${stats.totalCost}`)
console.log(`Average latency: ${stats.averageLatency}ms`)

stats.topPrompts.forEach(p => {
  console.log(`${p.promptName}: ${p.executionCount} executions, avg cost $${p.averageCost}`)
})

Python

Analyze Failed Executions

python

import requests
from datetime import datetime, timedelta

api_key = 'your-api-key'
endpoint = 'https://api.promptforge.sh/graphql'

def get_failed_executions(days: int = 7):
    query = """
        query GetExecutions($status: ExecutionStatus) {
          executions(status: $status) {
            id
            error
            createdAt
            promptVersion {
              prompt {
                name
              }
            }
          }
        }
    """

    response = requests.post(
        endpoint,
        headers={
            'Content-Type': 'application/json',
            'X-API-Key': api_key,
        },
        json={
            'query': query,
            'variables': {
                'status': 'error',
            },
        },
    )

    result = response.json()

    if 'errors' in result:
        raise Exception(result['errors'][0]['message'])

    return result['data']['executions']

# Usage
failed = get_failed_executions()

# Group errors by type
error_types = {}
for execution in failed:
    error_msg = execution['error']
    error_types[error_msg] = error_types.get(error_msg, 0) + 1

print(f"Total failed executions: {len(failed)}")
print("\nError breakdown:")
for error, count in sorted(error_types.items(), key=lambda x: x[1], reverse=True):
    print(f"  {error}: {count}")

Cost Analysis Script

Track Daily Costs

typescript

async function analyzeDailyCosts(days: number = 30) {
  const stats = await getExecutionStats(days)

  const dailyCosts = stats.executionsByDay.map(day => ({
    date: day.date,
    count: day.count,
    // Estimate daily cost based on total cost and execution count
    estimatedCost: (stats.totalCost / stats.totalExecutions) * day.count,
  }))

  console.log('Daily cost breakdown:')
  dailyCosts.forEach(day => {
    console.log(`${day.date}: ${day.count} executions, ~$${day.estimatedCost.toFixed(4)}`)
  })

  const avgDailyCost = dailyCosts.reduce((sum, d) => sum + d.estimatedCost, 0) / dailyCosts.length
  const projectedMonthlyCost = avgDailyCost * 30

  console.log(`\nAverage daily cost: $${avgDailyCost.toFixed(4)}`)
  console.log(`Projected monthly cost: $${projectedMonthlyCost.toFixed(2)}`)
}

analyzeDailyCosts()

Pagination

Use limit and offset parameters for pagination:

async function getAllExecutions() {
  const pageSize = 100
  let offset = 0
  let allExecutions = []

  while (true) {
    const response = await fetch(endpoint, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-API-Key': apiKey,
      },
      body: JSON.stringify({
        query: `
          query GetExecutions($limit: Int, $offset: Int) {
            executions(limit: $limit, offset: $offset) {
              id
              status
              costUsd
            }
          }
        `,
        variables: {
          limit: pageSize,
          offset: offset,
        },
      }),
    })

    const { data } = await response.json()
    const executions = data.executions

    if (executions.length === 0) {
      break
    }

    allExecutions.push(...executions)
    offset += pageSize
  }

  return allExecutions
}

Best Practices

Monitor Regularly - Check execution statistics weekly to catch issues early, identify cost spikes, and optimize performance.

Track Success Rates - Monitor the success rate metric. A drop below 95% may indicate integration issues or input validation problems.

Analyze Failed Executions - Group errors by type to identify systematic issues like API key problems or schema mismatches.

Use Time Filters - When querying large execution histories, always use date filters to improve performance and reduce response size.

Cache Statistics - Execution statistics change slowly. Cache results for 5-15 minutes to reduce API calls in dashboards.

Next Steps

Prompts API

Execute prompts and create new executions

Prompts API →

Monitoring Guide

Best practices for monitoring executions

Execution Concepts →