# WebView / native messages

Upload Tracking emits structured messages so the host app can react.

Transport depends on your integration.

In a browser iframe, it’s `postMessage`.

In mobile apps, it’s your WebView bridge.

Some bridges stringify the payload. Handle both forms.

### Typical V3 flow (Upload Tracking)

#### Video (`source=video`)

1. `initialization` while the page and video load.
2. `posture` until the exercise is ready.
3. During playback:
   * `counter`
   * optional `form_score` for built-in exercise tracking
   * optional `reference_score` inside `counter` for reference-based tracking
   * optional `progression`, `recommendations`
   * optional `keypoints` / `angles`
4. End of video:
   * `counter` with `final=true`
   * optional `exercise_summary` when using `reference`
   * optional `keypoints_batch` (if enabled)
   * optional `export_ready` (if `export=true`)

#### Image (`source=image`)

1. `initialization` while the image loads.
2. Optional `keypoints` / `angles`.
3. `image_overlay` with a base64 overlay image.

### Message types

#### `initialization` (video + image)

Fields:

* `type` (`string`) — `"initialization"`.
* `message` (`string`) — status.
* `ready` (`boolean`) — `true` when analysis can run.
* `source` (`string`, optional) — `"video"` or `"image"`.

Example:

```json
{
  "type": "initialization",
  "source": "video",
  "message": "Loading video",
  "ready": false
}
```

#### `environment` (video + image)

Sent after backend init.

Fields:

* `type` (`string`) — `"environment"`.
* `poseBackend` (`string`) — `"webgl"` or `"wasm"`.
* `webgl` (`boolean`) — whether WebGL is available.

Example:

```json
{
  "type": "environment",
  "poseBackend": "webgl",
  "webgl": true
}
```

#### `error` (video + image)

Fields:

* `type` (`string`) — `"error"`.
* `code` (`string`) — machine-readable error code.
* `message` (`string`) — human-readable description.
* `details` (`object`, optional).

Example:

```json
{
  "type": "error",
  "code": "video_load_error",
  "message": "Failed to load video",
  "details": {
    "videoSrc": "https://your-cdn.com/video.mp4"
  }
}
```

Common codes you should handle:

* `missing_token`
* `invalid_token`
* `invalid_exercise` (video only)
* `developer_features_not_allowed`
* `video_load_error`
* `image_load_error`
* `media_decode_error`
* `cross_origin_video`
* `cross_origin_image`
* `webgl_unavailable`
* `jump_analysis_missing_height` (for `exercise=jump_analysis`)

### Video-only messages (`source=video`)

#### `posture`

Same meaning as Tracking placement.

```json
{
  "type": "posture",
  "message": "Face the camera. Keep your full body in frame.",
  "direction": "in-frame",
  "ready": false,
  "requirements": ["full_body_visible"]
}
```

#### `counter`

Fields:

* `type` (`string`) — `"counter"`.
* `current_count` (`number`) — reps or seconds.
* `final` (`boolean`, optional) — `true` at end-of-video.
* `reference_score` (`object`, optional) — present when using `reference=REFERENCE_UUID`.

```json
{
  "type": "counter",
  "current_count": 12,
  "final": true
}
```

When using reference-based tracking, `reference_score` may contain:

* `overallScore`
* `poseScore`
* `timingScore`
* `movementScore`
* `grade`

```json
{
  "type": "counter",
  "current_count": 2,
  "reference_score": {
    "overallScore": 0.84,
    "poseScore": 0.79,
    "timingScore": 0.91,
    "movementScore": 0.85,
    "grade": "B"
  }
}
```

#### `exercise_summary` (reference tracking only)

When using `reference=REFERENCE_UUID`, Upload Tracking may emit a final `exercise_summary` event at the end of video analysis.

This event gives aggregate data for the full video.

Fields:

* `type` (`string`) — `"exercise_summary"`.
* `counter` (`number`) — total repetitions detected.
* `avg_similarity` (`number | null`) — average similarity across frames.
* `avg_rep_score` (`number | null`) — average repetition score across counted reps.
* `total_frames_compared` (`number`) — number of frames compared.

```json
{
  "type": "exercise_summary",
  "counter": 12,
  "avg_similarity": 0.76,
  "avg_rep_score": 0.81,
  "total_frames_compared": 530
}
```

#### `form_score` (optional)

```json
{
  "type": "form_score",
  "score": 76,
  "label": "good"
}
```

#### `progression` (plan-gated)

```json
{
  "type": "progression",
  "value": 63
}
```

#### `recommendations` (plan-gated)

```json
{
  "type": "recommendations",
  "data": ["Keep your back straight.", "Control the descent."]
}
```

#### `keypoints` (plan-gated)

```json
{
  "type": "keypoints",
  "data": [
    { "name": "nose", "x": 320.1, "y": 80.0, "score": 0.98 },
    { "name": "left_hip", "x": 345.4, "y": 265.2, "score": 0.96 }
  ]
}
```

#### `angles` (plan-gated)

```json
{
  "type": "angles",
  "data": {
    "left_side": { "knee_angle": { "from_hip_to_ankle": 162 } },
    "right_side": { "knee_angle": { "from_hip_to_ankle": 167 } }
  }
}
```

#### `keypoints_batch` (plan-gated)

Sent near the end of a video analysis.

It contains time-stamped keypoints for the whole clip.

Fields:

* `type` (`string`) — `"keypoints_batch"`.
* `fps` (`number`) — sampling rate.
* `frames` (`array<object>`) — per-frame keypoints.

Example (truncated):

```json
{
  "type": "keypoints_batch",
  "fps": 30,
  "frames": [
    {
      "t": 0,
      "keypoints": [{ "name": "nose", "x": 321.2, "y": 79.9, "score": 0.99 }]
    },
    {
      "t": 33,
      "keypoints": [{ "name": "nose", "x": 320.9, "y": 80.4, "score": 0.99 }]
    }
  ]
}
```

#### `export_ready` (plan-gated, `export=true`)

Sent when export finishes.

Fields:

* `type` (`string`) — `"export_ready"`.
* `url` (`string`) — a downloadable URL.
* `mimeType` (`string`, optional) — example: `"video/mp4"`.

Example:

```json
{
  "type": "export_ready",
  "url": "blob:https://app.posetracker.com/1c7c0a0b-0f1a-4cd6-8e8a-31b4f1b6c2b3",
  "mimeType": "video/mp4"
}
```

#### Jump messages (video, custom exercises)

These fire only for:

* `exercise=jump_analysis`
* `exercise=air_time_jump`

They can be emitted during analysis.

They are identical to the real-time message types.

**`jump_calibration` (jump\_analysis only)**

```json
{
  "type": "jump_calibration",
  "ready": false,
  "message": "Hold still"
}
```

**`jump_started`**

```json
{
  "type": "jump_started"
}
```

**`jump_discarded`**

```json
{
  "type": "jump_discarded",
  "reason": "unstable_baseline"
}
```

**`jump_height`**

```json
{
  "type": "jump_height",
  "jumpHeightCm": 33.7,
  "final": true
}
```

**`jump_summary`**

Sent near the end of a video analysis.

```json
{
  "type": "jump_summary",
  "totalJumps": 3,
  "avgJumpHeight": 31.2,
  "maxJumpHeight": 35.4,
  "minJumpHeight": 27.8,
  "final": true
}
```

### Image-only messages (`source=image`)

#### `image_overlay`

Sent after processing completes.

Fields:

* `type` (`string`) — `"image_overlay"`.
* `dataUrl` (`string`) — base64 PNG or JPEG.

```json
{
  "type": "image_overlay",
  "dataUrl": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA..."
}
```

#### `keypoints` and `angles` (plan-gated)

For images, these are emitted once.