# SEE2AI 高级视频理解

> SEE2AI advanced video understanding: jointly analyzes picture and embedded audio, returning a structured summary, answer, timeline, observations, and best-effort transcript. The existing video-understanding Action is unchanged.

## 📖 接入指引

**Endpoint**: `POST /api/v1/actions/see2ai_video_understanding_v2`

**认证方式**: Bearer Token (租户 API Key)

### Endpoints

| Method | Path | Purpose |
| --- | --- | --- |
| `POST` | `/api/v1/actions/see2ai_video_understanding_v2` | Submit the standard action request. |

> **⚠️ 重要提示**
> 
> 本 API 是 SEE2AI 平台标准化封装后的接口，**参数格式与任何上游 API 不同**。
> 请直接使用下方 Schema 中定义的参数，无需参考其他文档。
> 
> *例如：*
> *- 文本模型参数可能是 `query` 而不是 `messages` 数组*
> *- 视频模型参数可能是 `brief` (创意简报) 而不是复杂的配置项*

## 📥 输入参数 (Input Schema)

```json
{
  "description": "视频语义理解输入（含输出形态选择）。",
  "properties": {
    "video_url": {
      "description": "视频公网 URL（http/https，mp4/avi/mov，≤50MB）。只接受公网 URL；本地文件请先调用 `see2ai_storage_upload_v1` 上传后再填入返回的 URL。",
      "title": "Video Url",
      "type": "string"
    },
    "query": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "要问视频的问题（可选）。缺省时执行全面理解（摘要+时间线+观察）。",
      "title": "Query"
    },
    "analysis_mode": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": "full",
      "description": "分析模式：summary（只要摘要）/ qa（回答 query，必须提供 query）/ timeline（侧重时间线）/ full（全量结构化，默认）。",
      "title": "Analysis Mode"
    },
    "audio_mode": {
      "default": "native",
      "description": "音频处理：native（默认，同时理解画面与视频自带音频——语音/旁白/音乐）；off（忽略音轨，只析画面）。",
      "title": "Audio Mode",
      "type": "string"
    },
    "return_timeline": {
      "default": true,
      "description": "是否返回 key_moments 时间线（按秒级时间戳的关键节点列表）。",
      "title": "Return Timeline",
      "type": "boolean"
    },
    "fps": {
      "anyOf": [
        {
          "type": "number"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "画面抽帧率（可选，0.2-5，缺省用上游默认 1）。画面变化剧烈时调高、静态画面调低以节省词元。",
      "title": "Fps"
    },
    "enable_thinking": {
      "default": false,
      "description": "是否启用深度思考模式（默认关闭：会增加时延与词元消耗，且与视频帧共享输入预算）。",
      "title": "Enable Thinking",
      "type": "boolean"
    },
    "business_id": {
      "anyOf": [
        {
          "maxLength": 128,
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Optional stable business id for atomic retry protection. When the X-Idempotency-Key header is also supplied, both values must match.",
      "title": "Business Id"
    },
    "observation_fallback": {
      "anyOf": [
        {
          "const": "stronger_model",
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Optional. When set to \"stronger_model\", if the structured video observation (timeline and audio evidence) fails upstream, it is retried once with a stronger model. Extra usage is billed at actual usage and never exceeds the confirmed spending cap. Omit to keep the default behavior.",
      "title": "Observation Fallback"
    },
    "output_profile": {
      "default": "envelope",
      "description": "输出形态：envelope（默认，answer 按 query 要求的格式回写）/ director（平台自持的导演级脚本，query 退化为补充上下文，结果另以 director_script 类型化字段返回）。",
      "enum": [
        "envelope",
        "director"
      ],
      "title": "Output Profile",
      "type": "string"
    }
  },
  "required": [
    "video_url"
  ],
  "title": "VideoUnderstandingV2Input",
  "type": "object"
}
```

### 参数说明

| 参数名 | 类型 | 必填 | 默认值 | 说明 |
|--------|------|------|--------|------|
| `video_url` | string | ✅ 是 | `-` | 视频公网 URL（http/https，mp4/avi/mov，≤50MB）。只接受公网 URL；本地文件请先调用 `see2ai_storage_upload_v1` 上传后再填入返回的 URL。 |
| `query` | string / null | ❌ 否 | `-` | 要问视频的问题（可选）。缺省时执行全面理解（摘要+时间线+观察）。 |
| `analysis_mode` | string / null | ❌ 否 | `full` | 分析模式：summary（只要摘要）/ qa（回答 query，必须提供 query）/ timeline（侧重时间线）/ full（全量结构化，默认）。 |
| `audio_mode` | string | ❌ 否 | `native` | 音频处理：native（默认，同时理解画面与视频自带音频——语音/旁白/音乐）；off（忽略音轨，只析画面）。 |
| `return_timeline` | boolean | ❌ 否 | `True` | 是否返回 key_moments 时间线（按秒级时间戳的关键节点列表）。 |
| `fps` | number / null | ❌ 否 | `-` | 画面抽帧率（可选，0.2-5，缺省用上游默认 1）。画面变化剧烈时调高、静态画面调低以节省词元。 |
| `enable_thinking` | boolean | ❌ 否 | `False` | 是否启用深度思考模式（默认关闭：会增加时延与词元消耗，且与视频帧共享输入预算）。 |
| `business_id` | string / null | ❌ 否 | `-` | Optional stable business id for atomic retry protection. When the X-Idempotency-Key header is also supplied, both values must match. |
| `observation_fallback` | string / null | ❌ 否 | `-` | Optional. When set to "stronger_model", if the structured video observation (timeline and audio evidence) fails upstream, it is retried once with a stronger model. Extra usage is billed at actual usage and never exceeds the confirmed spending cap. Omit to keep the default behavior. |
| `output_profile` | string enum | ❌ 否 | `envelope` | 输出形态：envelope（默认，answer 按 query 要求的格式回写）/ director（平台自持的导演级脚本，query 退化为补充上下文，结果另以 director_script 类型化字段返回）。 |

## 📤 输出结果 (Output Schema)

```json
{
  "$defs": {
    "DirectorAsset": {
      "properties": {
        "ref": {
          "description": "本片段内的素材引用，如 @图1。",
          "title": "Ref",
          "type": "string"
        },
        "global_name": {
          "description": "跨片段复用时的全局素材名。",
          "title": "Global Name",
          "type": "string"
        },
        "usage": {
          "description": "该素材在本片段的用途与要求。",
          "title": "Usage",
          "type": "string"
        }
      },
      "required": [
        "ref",
        "global_name",
        "usage"
      ],
      "title": "DirectorAsset",
      "type": "object"
    },
    "DirectorOverview": {
      "properties": {
        "total_shots": {
          "anyOf": [
            {
              "type": "integer"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Total Shots"
        },
        "total_duration_seconds": {
          "anyOf": [
            {
              "type": "number"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Total Duration Seconds"
        },
        "aspect_ratio": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Aspect Ratio"
        },
        "style_tone": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Style Tone"
        },
        "audio_bed": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Audio Bed"
        },
        "compliance": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Compliance"
        },
        "recommended_model": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "title": "Recommended Model"
        }
      },
      "title": "DirectorOverview",
      "type": "object"
    },
    "DirectorScript": {
      "description": "Typed director script; callers never parse JSON out of a string.",
      "properties": {
        "overview": {
          "$ref": "#/$defs/DirectorOverview"
        },
        "global_assets": {
          "items": {
            "additionalProperties": true,
            "type": "object"
          },
          "title": "Global Assets",
          "type": "array"
        },
        "segments": {
          "items": {
            "$ref": "#/$defs/DirectorSegment"
          },
          "title": "Segments",
          "type": "array"
        }
      },
      "title": "DirectorScript",
      "type": "object"
    },
    "DirectorSegment": {
      "properties": {
        "index": {
          "description": "片段序号。",
          "title": "Index",
          "type": "integer"
        },
        "entry": {
          "description": "Seedance 入口：首尾帧 或 全能参考。",
          "title": "Entry",
          "type": "string"
        },
        "duration_seconds": {
          "description": "本片段生成时长，4–15 秒。",
          "title": "Duration Seconds",
          "type": "number"
        },
        "assets": {
          "items": {
            "$ref": "#/$defs/DirectorAsset"
          },
          "title": "Assets",
          "type": "array"
        },
        "shots": {
          "items": {
            "$ref": "#/$defs/DirectorShot"
          },
          "title": "Shots",
          "type": "array"
        },
        "negative": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "本片段的负向排除词。",
          "title": "Negative"
        }
      },
      "required": [
        "index",
        "entry",
        "duration_seconds"
      ],
      "title": "DirectorSegment",
      "type": "object"
    },
    "DirectorShot": {
      "properties": {
        "index": {
          "description": "镜头序号，按时间轴升序。",
          "title": "Index",
          "type": "integer"
        },
        "start_second": {
          "description": "起始秒。",
          "title": "Start Second",
          "type": "number"
        },
        "end_second": {
          "description": "结束秒。",
          "title": "End Second",
          "type": "number"
        },
        "duration_seconds": {
          "description": "该镜头时长。",
          "title": "Duration Seconds",
          "type": "number"
        },
        "prompt": {
          "description": "一段吃满 15 维的导演视角自然语言。",
          "title": "Prompt",
          "type": "string"
        },
        "safe_area": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ],
          "default": null,
          "description": "画幅与安全区约束。",
          "title": "Safe Area"
        }
      },
      "required": [
        "index",
        "start_second",
        "end_second",
        "duration_seconds",
        "prompt"
      ],
      "title": "DirectorShot",
      "type": "object"
    },
    "KeyMoment": {
      "description": "时间线关键节点",
      "properties": {
        "start_second": {
          "description": "起始秒",
          "title": "Start Second",
          "type": "number"
        },
        "end_second": {
          "description": "结束秒",
          "title": "End Second",
          "type": "number"
        },
        "title": {
          "description": "节点标题",
          "title": "Title",
          "type": "string"
        },
        "description": {
          "description": "节点描述",
          "title": "Description",
          "type": "string"
        }
      },
      "required": [
        "start_second",
        "end_second",
        "title",
        "description"
      ],
      "title": "KeyMoment",
      "type": "object"
    },
    "VideoUnderstandingUsage": {
      "description": "词元使用统计",
      "properties": {
        "input_tokens": {
          "default": 0,
          "title": "Input Tokens",
          "type": "integer"
        },
        "output_tokens": {
          "default": 0,
          "title": "Output Tokens",
          "type": "integer"
        },
        "total_tokens": {
          "default": 0,
          "title": "Total Tokens",
          "type": "integer"
        }
      },
      "title": "VideoUnderstandingUsage",
      "type": "object"
    }
  },
  "description": "Additive output schema describing the composed audio path precisely.",
  "properties": {
    "input_tokens": {
      "default": 0,
      "description": "输入消耗的词元数量",
      "title": "Input Tokens",
      "type": "integer"
    },
    "output_tokens": {
      "default": 0,
      "description": "输出消耗的词元数量",
      "title": "Output Tokens",
      "type": "integer"
    },
    "total_tokens": {
      "default": 0,
      "description": "总消耗的词元数量",
      "title": "Total Tokens",
      "type": "integer"
    },
    "model": {
      "default": "",
      "description": "实际使用的 SEE2AI 化身模型名称（由各 action 显式填写）",
      "title": "Model",
      "type": "string"
    },
    "source": {
      "default": "see2ai",
      "description": "服务来源（统一 see2ai；真实路由仅平台内部可见）",
      "title": "Source",
      "type": "string"
    },
    "cost_points": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "预先计算的扣费词元（如果插件内部已经计算好费用，则直接使用此值，忽略基于词元的计算）",
      "title": "Cost Points"
    },
    "summary": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "视频整体摘要（音画综合）",
      "title": "Summary"
    },
    "answer": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "针对 query 的直接回答（无 query 时为 null）",
      "title": "Answer"
    },
    "key_moments": {
      "description": "时间线关键节点（秒级）",
      "items": {
        "$ref": "#/$defs/KeyMoment"
      },
      "title": "Key Moments",
      "type": "array"
    },
    "visual_observations": {
      "description": "画面侧关键观察",
      "items": {
        "type": "string"
      },
      "title": "Visual Observations",
      "type": "array"
    },
    "audio_transcript": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "音频中语音内容的转述（模型 best-effort；无有效语音或 audio_mode=off 时为 null）",
      "title": "Audio Transcript"
    },
    "audio_observations": {
      "description": "音轨侧语义观察",
      "items": {
        "type": "string"
      },
      "title": "Audio Observations",
      "type": "array"
    },
    "audio_status": {
      "default": "composed",
      "description": "Audio path: composed (dedicated embedded-audio branch), off (caller disabled audio), or unavailable (visual-only partial result).",
      "title": "Audio Status",
      "type": "string"
    },
    "thinking_content": {
      "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "深度思考过程内容（如果启用）",
      "title": "Thinking Content"
    },
    "usage": {
      "anyOf": [
        {
          "$ref": "#/$defs/VideoUnderstandingUsage"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "API 使用量统计"
    },
    "provider_config_id": {
      "anyOf": [
        {
          "type": "integer"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "请求追踪 ID（便于问题排查）",
      "title": "Provider Config Id"
    },
    "warnings": {
      "anyOf": [
        {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "Non-blocking notes.",
      "title": "Warnings"
    },
    "director_script": {
      "anyOf": [
        {
          "$ref": "#/$defs/DirectorScript"
        },
        {
          "type": "null"
        }
      ],
      "default": null,
      "description": "导演级脚本（仅 output_profile=director 时返回），调用方无需解析字符串。"
    }
  },
  "title": "VideoUnderstandingV2Output",
  "type": "object"
}
```

### 输出字段说明

| 字段名 | 类型 | 说明 |
|--------|------|------|
| `input_tokens` | integer | 输入消耗的词元数量 |
| `output_tokens` | integer | 输出消耗的词元数量 |
| `total_tokens` | integer | 总消耗的词元数量 |
| `model` | string | 实际使用的 SEE2AI 化身模型名称（由各 action 显式填写） |
| `source` | string | 服务来源（统一 see2ai；真实路由仅平台内部可见） |
| `cost_points` | integer / null | 预先计算的扣费词元（如果插件内部已经计算好费用，则直接使用此值，忽略基于词元的计算） |
| `summary` | string / null | 视频整体摘要（音画综合） |
| `answer` | string / null | 针对 query 的直接回答（无 query 时为 null） |
| `key_moments` | array | 时间线关键节点（秒级） |
| `visual_observations` | array | 画面侧关键观察 |
| `audio_transcript` | string / null | 音频中语音内容的转述（模型 best-effort；无有效语音或 audio_mode=off 时为 null） |
| `audio_observations` | array | 音轨侧语义观察 |
| `audio_status` | string | Audio path: composed (dedicated embedded-audio branch), off (caller disabled audio), or unavailable (visual-only partial result). |
| `thinking_content` | string / null | 深度思考过程内容（如果启用） |
| `usage` | object / null | API 使用量统计 |
| `provider_config_id` | integer / null | 请求追踪 ID（便于问题排查） |
| `warnings` | array / null | Non-blocking notes. |
| `director_script` | object / null | 导演级脚本（仅 output_profile=director 时返回），调用方无需解析字符串。 |

## 💡 调用示例

### cURL 示例

```bash
curl -X POST "https://see2ai.com/api/v1/actions/see2ai_video_understanding_v2" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer sk-your-tenant-api-key" \
     -d '{
  "video_url": "https://example.com/video.mp4",
  "query": "Explain quantum entanglement in simple terms.",
  "analysis_mode": "full",
  "audio_mode": "native",
  "return_timeline": true,
  "enable_thinking": false,
  "output_profile": "envelope"
}'
```

### Python 示例

```python
import requests

API_KEY = "sk-your-tenant-api-key"
ENDPOINT = "https://see2ai.com/api/v1/actions/see2ai_video_understanding_v2"

headers = {
    "Content-Type": "application/json",
    "Authorization": f"Bearer {API_KEY}"
}

payload = {
  "video_url": "https://example.com/video.mp4",
  "query": "Explain quantum entanglement in simple terms.",
  "analysis_mode": "full",
  "audio_mode": "native",
  "return_timeline": true,
  "enable_thinking": false,
  "output_profile": "envelope"
}

response = requests.post(ENDPOINT, json=payload, headers=headers)
result = response.json()

print(f"结果：{result}")
```

## 💰 计费说明

> Metered from the exact combined input/output usage of the observation branch and, when a question is asked, the answering branch.

**注意**: 所有费用都以词元为单位，系统会自动从租户余额中扣除。

## ⚠️ 常见错误码

| HTTP 状态码 | 错误原因 | 解决方案 |
| --- | --- | --- |
| `401` | API Key 无效或租户余额不足 | 检查 API Key 是否正确，或联系管理员充值 |
| `413` | 视频超过 50MB 或输入词元超限 | 请稍后重试 |
| `415` | video_url 指向音频或不支持的容器格式 | 请稍后重试 |
| `422` | 输入参数格式错误（未通过校验） | 检查请求参数是否符合 Schema 要求 |
| `429` | 调用频率超限 | 降低请求频率，或联系管理员调整限流配置 |
| `502` | 上游视频理解服务错误 | 请稍后重试 |
| `504` | 上游服务响应超时 | 请稍后重试 |

## 📞 技术支持

- **文档问题**: 查看 [SEE2AI 官方文档](https://see2ai.com/docs)
- **API 状态**: 访问 [服务状态页面](https://see2ai.com/status)
- **问题反馈**: 联系技术支持团队

---

**API 版本**: see2ai_video_understanding_v2
**最后更新**: 2026-03-23