Integrating with AI Agents

The Agent Workflow

A CortexUI-integrated agent follows a structured workflow that replaces visual heuristics with contract queries at every step:

1. getScreenContext()     → Where am I? What entity am I looking at?
2. getAvailableActions()  → What can I do from here?
3. getFormSchema(id)      → What fields does this form require?
4. [Interact]             → Fill fields, click actions
5. getRecentEvents()      → Did the action complete? What was the result?

Each step queries the runtime API directly — no DOM parsing, no visual inference, no heuristics. The contract tells the agent everything it needs to act with confidence.

The Runtime API

The CortexUI runtime exposes a global API on window.__CORTEX_UI__ that agents access via page.evaluate():

// Get the current screen context
window.__CORTEX_UI__.getScreenContext()
// Returns: { screenId, screenName, entity?, entityId?, sections: [...] }

// Get all available (non-disabled) actions
window.__CORTEX_UI__.getAvailableActions()
// Returns: Array<{ id, action, state, entity?, entityId?, section?, description? }>

// Get the schema for a specific form
window.__CORTEX_UI__.getFormSchema('profile-form')
// Returns: { formId, fields: Array<{ id, type, label, required, currentValue }> }

// Get visible entities on the current screen
window.__CORTEX_UI__.getVisibleEntities()
// Returns: Array<{ entity, entityId, section, actions: [...] }>

// Get recent events from the event log
window.__CORTEX_UI__.getRecentEvents()
// Returns: Array<{ type, actionId, result?, message?, timestamp }>

Example: Claude Agent Integration

This example shows a complete CortexUI-aware agent step using the Anthropic Claude API. The agent reads the UI contract, sends it to Claude for decision-making, executes the indicated action, and verifies the result.

import Anthropic from '@anthropic-ai/sdk';
import type { Page } from 'playwright';

const claude = new Anthropic();

// A CortexUI-aware agent step using the Claude API
async function executeCortexUIAction(page: Page, instruction: string) {
  // Step 1: Get UI context from the contract
  const context = await page.evaluate(() => ({
    screen: window.__CORTEX_UI__.getScreenContext(),
    actions: window.__CORTEX_UI__.getAvailableActions(),
    entities: window.__CORTEX_UI__.getVisibleEntities()
  }));

  // Step 2: Pass context to Claude for action selection
  const response = await claude.messages.create({
    model: 'claude-opus-4-6',
    max_tokens: 256,
    messages: [{
      role: 'user',
      content: `
        Current UI Context:
        ${JSON.stringify(context, null, 2)}

        Instruction: ${instruction}

        Based on the available actions in the UI context, which action ID should
        I click to fulfill this instruction? Respond with just the action ID string,
        nothing else. If no action matches, respond with "NO_ACTION".
      `
    }]
  });

  const actionId = response.content[0].type === 'text'
    ? response.content[0].text.trim()
    : null;

  if (!actionId || actionId === 'NO_ACTION') {
    throw new Error(`No matching action found for instruction: "${instruction}"`);
  }

  // Step 3: Validate the action before executing
  const action = context.actions.find(a => a.id === actionId);
  if (!action) {
    throw new Error(`Action "${actionId}" not found in available actions`);
  }
  if (action.state === 'disabled') {
    throw new Error(`Action "${actionId}" is currently disabled`);
  }

  // Step 4: Execute the action
  await page.click(`[data-ai-id="${actionId}"]`);

  // Step 5: Wait for and verify completion
  const result = await waitForActionCompletion(page, actionId, 10_000);
  return result;
}

async function waitForActionCompletion(page: Page, actionId: string, timeoutMs: number) {
  const start = Date.now();

  while (Date.now() - start < timeoutMs) {
    const events = await page.evaluate((id) =>
      window.__CORTEX_UI__.getRecentEvents()
        .filter(e => e.type === 'action_completed' && e.actionId === id),
      actionId
    );

    if (events.length > 0) {
      return events[events.length - 1];
    }

    await page.waitForTimeout(100);
  }

  throw new Error(`Action "${actionId}" did not complete within ${timeoutMs}ms`);
}

Form Filling Workflow

For form interactions, query the form schema before filling to know exactly what fields are available and what types they expect:

async function fillAndSubmitForm(page: Page, formId: string, data: Record<string, string>) {
  // Get the form schema
  const schema = await page.evaluate((id) =>
    window.__CORTEX_UI__.getFormSchema(id),
    formId
  );

  // Validate that all required fields are provided
  const missingRequired = schema.fields
    .filter(f => f.required && !data[f.id])
    .map(f => f.label);

  if (missingRequired.length > 0) {
    throw new Error(`Missing required fields: ${missingRequired.join(', ')}`);
  }

  // Fill each field
  for (const field of schema.fields) {
    if (data[field.id] !== undefined) {
      await page.fill(`[data-ai-id="${field.id}"]`, data[field.id]);
    }
  }

  // Find and click the form's submit action
  const submitAction = await page.evaluate((id) =>
    window.__CORTEX_UI__.getAvailableActions()
      .find(a => a.formId === id && a.action.startsWith('submit')),
    formId
  );

  if (!submitAction) {
    throw new Error(`No submit action found for form "${formId}"`);
  }

  await page.click(`[data-ai-id="${submitAction.id}"]`);

  const result = await waitForActionCompletion(page, submitAction.id, 15_000);

  if (result.result === 'error') {
    throw new Error(`Form submission failed: ${result.message}`);
  }

  return result;
}

Error Handling

Robust agent implementations handle the full range of failure modes:

async function safeExecuteAction(actionId: string): Promise<ActionResult> {
  // Pre-flight: verify action is available
  const actions = window.__CORTEX_UI__.getAvailableActions();
  const action = actions.find(a => a.id === actionId);

  if (!action) {
    throw new Error(
      `Action "${actionId}" not found. Available actions: ${actions.map(a => a.id).join(', ')}`
    );
  }

  if (action.state === 'disabled') {
    throw new Error(
      `Action "${actionId}" is currently disabled and cannot be executed.`
    );
  }

  if (action.state === 'loading') {
    throw new Error(
      `Action "${actionId}" is already in progress. Wait for it to complete before retrying.`
    );
  }

  // Execute
  const el = document.querySelector(`[data-ai-id="${actionId}"]`);
  if (!el) {
    throw new Error(`Element for action "${actionId}" not found in DOM.`);
  }
  (el as HTMLElement).click();

  // Wait for result
  const result = await waitForActionResult(actionId, 10_000);

  if (result.result === 'error') {
    throw new Error(`Action "${actionId}" failed: ${result.message ?? 'Unknown error'}`);
  }

  return result;
}

async function waitForActionResult(actionId: string, timeoutMs: number): Promise<ActionResult> {
  return new Promise((resolve, reject) => {
    const deadline = setTimeout(() => {
      reject(new Error(`Timeout waiting for action "${actionId}" to complete after ${timeoutMs}ms`));
    }, timeoutMs);

    const poll = setInterval(() => {
      const events = window.__CORTEX_UI__.getRecentEvents();
      const completed = events.find(
        e => e.type === 'action_completed' && e.actionId === actionId
      );

      if (completed) {
        clearInterval(poll);
        clearTimeout(deadline);
        resolve(completed);
      }
    }, 100);
  });
}

Integrating with Playwright

For browser automation with Playwright, wrap the runtime API in a helper class:

import type { Page } from 'playwright';

export class CortexUIPage {
  constructor(private page: Page) {}

  async getScreenContext() {
    return this.page.evaluate(() => window.__CORTEX_UI__.getScreenContext());
  }

  async getAvailableActions() {
    return this.page.evaluate(() => window.__CORTEX_UI__.getAvailableActions());
  }

  async getFormSchema(formId: string) {
    return this.page.evaluate((id) => window.__CORTEX_UI__.getFormSchema(id), formId);
  }

  async clickAction(actionId: string) {
    await this.page.click(`[data-ai-id="${actionId}"]`);
  }

  async waitForAction(actionId: string, timeoutMs = 10_000) {
    const start = Date.now();
    while (Date.now() - start < timeoutMs) {
      const events = await this.page.evaluate(
        (id) => window.__CORTEX_UI__.getRecentEvents().filter(
          e => e.type === 'action_completed' && e.actionId === id
        ),
        actionId
      );
      if (events.length > 0) return events[0];
      await this.page.waitForTimeout(100);
    }
    throw new Error(`Action "${actionId}" timed out`);
  }

  async executeAction(actionId: string) {
    await this.clickAction(actionId);
    return this.waitForAction(actionId);
  }
}

// Usage:
const ui = new CortexUIPage(page);
const context = await ui.getScreenContext();
await ui.executeAction('save-profile');