VLM-QA Evaluation Protocol

HOI-QA50-Strict-Proxy · Question bank, prompts, sampling, and scoring

Download full protocol ↓

1. Overview

This release documents the independently specified HOI-QA50 strict proxy used for the reported evaluation. It evaluates interaction mechanics through 50 defect questions across four categories. All scores are on a 0–100 scale; higher is better.

The four evaluation categories, question counts and score weights
CategoryQuestion IDsCountWeight
C · Action correctness and completionC01–C101025%
S · Contact, support, and spatial relationsS01–S161635%
M · Hand-object motion synchronization and timingM01–M121225%
A · Hand, limb, and object structureA01–A121215%

The 50 questions define the metric’s question bank. Applicability is assessed separately for each requested interaction and then shared across the methods being compared.

For each case, applicability is determined from the requested action, object label, and optional product reference before generated outputs are inspected. The same frozen profile is used across methods. Each applicable question is judged independently. Motion synchronization refers to hand–object motion coupling. Aesthetics, identity/reference similarity, background consistency, and audio quality are outside this metric's scope.

2. Scoring and critical-failure rules

Answer states

StateMeaningCredit
CLEANThe applicability condition is true and sufficient visible evidence confirms that the FAIL condition does not occur.1
DEFECTThe applicability condition is true and visible evidence satisfies the FAIL condition.0
UNCERTAINThe applicability condition is true, but occlusion, resolution, sampling, or conflicting evidence prevents a reliable decision; this is not scored as a pass.0
NOT_APPLICABLEThe applicability condition is false; exclude this question from the denominator.Excluded

Generated blur, cropping, occlusion, missing hands, or action failure cannot retroactively make an applicable question N/A. If relevant evidence is insufficient, the judge returns UNCERTAIN, which receives zero credit. NOT_APPLICABLE is assigned by the frozen profile and is forbidden as a response to an applicable judging request.

Per-video aggregation

Rik = 100 × CLEANik / (CLEANik + DEFECTik + UNCERTAINik)
Fik = min(Rik, every triggered category cap)
Hi = 1 / Σk (w̃k / Fik)
Qi = min(Hi, every triggered overall cap)
Dataset score = meani(Qi)

If every question on a category is N/A, that category is excluded and its weight is redistributed proportionally over the remaining categories (w̃). Any zero-valued valid category yields an overall score of zero. The implementation returns zero when no categories are valid. Dataset category scores are meani(Fik); substituting category means into the harmonic mean does not reproduce the dataset overall score.

Seven critical-failure caps

For each row, any listed question judged DEFECT triggers the rule. Multiple triggered caps are combined by taking the strictest bound.

Failure typeTrigger question IDsCategory capOverall cap
Wrong or missing primary actionC01, C02, C03C ≤ 035
Missing required final stateC04, C10C ≤ 2550
Visible contact gapS02S ≤ 5060
Detached or floating objectS01, S03, S04, S05, S10, S12, S16S ≤ 2540
Penetration or fusionS06, S07, S08S ≤ 2045
Gross motion desynchronizationM01, M02, M03, M04, M07, M10, M12M ≤ 2045
Catastrophic structural failureA04, A05, A06, A07, A08, A09, A10A ≤ 2050

The exact failure thresholds are the question-specific FAIL conditions in the question bank and the caps above. They are not calibrated pixel-distance, physical-distance, or fixed-duration cutoffs. For S02, at least one clear hand–object air gap while contact is required triggers a defect. Brief but clear critical failures are not ignored. A C-category cap of zero already forces the harmonic score to zero, regardless of its accompanying overall cap.

3. Frame sampling and judge settings

Two visual input channels

  1. Full-video channel. The complete generated video is passed to the local multimodal endpoint with fps=16.0 and do_sample_frames=true. This is the requested processor sampling rate; the saved records do not expose every frame ultimately retained inside the model processor.
  2. Explicit audit-frame channel. ffprobe provides source frame count N and FPS f. Seventeen frames, including both endpoints, are selected with round(j × (N − 1) / 16) for j = 0,…,16, using Python's rounding behavior. Their timestamps are frame_index / f. Frames are labeled F01–F17 and timestamped using ffmpeg.

The overview has 5 columns with 192-pixel-wide tiles. Contact and structure questions (S/A) additionally receive four horizontal strips from these same 17 frames: 5 + 5 + 5 + 2 frames, each resized to 384 pixels wide, using JPEG qscale 2. Action and motion questions (C/M) receive the full video and overview. Optional product references supply identity/part context. The strips are full-frame arrangements, not automatically detected hand–object crops.

Exact 17-frame example: 80 frames, 25 FPS, 3.2 seconds
Frame IDSource indexTime (s)
F0100.000
F0250.200
F03100.400
F04150.600
F05200.800
F06251.000
F07301.200
F08351.400
F09401.600
F10441.760
F11491.960
F12542.160
F13592.360
F14642.560
F15692.760
F16742.960
F17793.160

Finite sampling can miss brief defects; the full-video channel and UNCERTAIN rule do not guarantee their elimination.

Recorded judge configuration

SettingRecorded value
Judge / backendQwen3.8-27B / qwen-local
Requested full-video sampling16 FPS; do_sample_frames=true
Temperature / top-p / top-k0.7 / 0.8 / 20
min-p0.0
Presence / repetition penalty1.5 / 1.0
Seed0
enable_thinking / preserve_thinkingfalse / false
Maximum output tokens8192
Backend retriesUp to 3 retries beyond the initial request
Evidence-validation retriesUp to 2 corrections beyond the initial judgment
JSON modeauto

Backend retries handle request/decoding failures. Evidence-validation retries handle malformed question IDs, fields, states, or evidence. JSON mode auto can fall back from enforced schema to prompt-constrained JSON. Valid judgments are not repeatedly sampled for majority voting or favorable selection; a valid UNCERTAIN remains zero credit.

4. Exact prompts and response schema

The exact bilingual judging request, pre-video profile prompt, and structured response formats are included in the documentation download. Each applicable question receives a separate judging call containing the request, object label, frozen action profile, applicability rationale, exact question and FAIL condition, critical flag, and fixed frame/time mapping.

Judging system prompt — exact text
You are a strict, evidence-first judge of one atomic human-object-interaction defect question.
The question's semantic applicability was frozen before the generated video was viewed. Therefore NOT_APPLICABLE is forbidden in this request.

Inspect the complete video in temporal order, the timestamped overview, and every supplied high-detail frame strip. The target-object reference image is identity/part context only and is never evidence that generated contact or motion is correct.

State rules:
- DEFECT: visible generated-video evidence satisfies the supplied FAIL condition.
- CLEAN: positive inspection of the relevant phase establishes that the FAIL condition does not occur. Mere presence of a person/object, intended action, proximity, or an unverified tiny image is not enough.
- UNCERTAIN: the question is applicable, but crop, blur, occlusion, sampling, or conflicting evidence prevents a reliable decision. Uncertainty receives zero credit and must not be converted to CLEAN.

For hand-object contact, CLEAN requires inspecting actual boundaries: nearby is not contact, and a visible air gap is a defect when contact/support is required. For temporal questions, inspect motion rather than inferring it from a single frame. Cite real timestamps and F01-F17 frame IDs. Reasons must describe visible pixels/motion, not restate the prompt. Return exactly one judgment for the supplied QID.
Pre-video applicability system prompt — exact text
You create a PRE-VIDEO action and applicability profile for strict human-object-interaction evaluation.
Use only the requested text, object label, and optional target-object reference image. You must not inspect, anticipate, or adapt to any generated video.

Action rules:
- Preserve the most specific requested action verb and its defining event. Never replace applying, brushing, curling, spraying, pouring, opening, pressing, wearing, or another functional action with generic holding/displaying merely because the tool must be held.
- Record a concrete, visually judgeable functional relation when one is needed: e.g. brush bristles contact/move across the face; hair enters or wraps around a curling wand's working region.
- A singular target object normally means expected_object_count=1. Do not infer two objects merely because two hands are visible in a reference.

Applicability rules:
- Decide semantic applicability before any generated output exists. Set applicable=false only when the requested action makes the question genuinely irrelevant.
- Do not use future crop, blur, occlusion, missing hands, or a bad generated trajectory to make a question inapplicable. If a semantically relevant property later cannot be observed, the video judge must answer UNCERTAIN.
- A question that detects an unexpected contact break, unsupported release, disappearance, or duplication remains applicable whenever that failure could violate the requested action, even if the request does not ask for the failure event.
- C02 is always applicable. Questions about the named target and its disappearance are applicable whenever a target object is specified.
Return exactly one applicability decision for every supplied QID, in QID order.
Pre-video profile user prompt — complete example
SYSTEM
You create a PRE-VIDEO action and applicability profile for strict human-object-interaction evaluation.
Use only the requested text, object label, and optional target-object reference image. You must not inspect, anticipate, or adapt to any generated video.

Action rules:
- Preserve the most specific requested action verb and its defining event. Never replace applying, brushing, curling, spraying, pouring, opening, pressing, wearing, or another functional action with generic holding/displaying merely because the tool must be held.
- Record a concrete, visually judgeable functional relation when one is needed: e.g. brush bristles contact/move across the face; hair enters or wraps around a curling wand's working region.
- A singular target object normally means expected_object_count=1. Do not infer two objects merely because two hands are visible in a reference.

Applicability rules:
- Decide semantic applicability before any generated output exists. Set applicable=false only when the requested action makes the question genuinely irrelevant.
- Do not use future crop, blur, occlusion, missing hands, or a bad generated trajectory to make a question inapplicable. If a semantically relevant property later cannot be observed, the video judge must answer UNCERTAIN.
- A question that detects an unexpected contact break, unsupported release, disappearance, or duplication remains applicable whenever that failure could violate the requested action, even if the request does not ask for the failure event.
- C02 is always applicable. Questions about the named target and its disappearance are applicable whenever a target object is specified.
Return exactly one applicability decision for every supplied QID, in QID order.

USER
Create a frozen action/applicability profile for this requested video.

Object label: bottle
Requested prompt:
The young woman with light brown hair pulled back stands in a brightly lit room with white walls, holding a small, light green dropper bottle labeled HOIVG-Bench. The product is a small-size bottle, compact enough to be comfortably held in one hand. She gestures expressively with her hands while demonstrating and speaking directly to the camera, showing how to use the bottle and engaging the viewer with her presentation.

For each question below, decide applicability from the request only. The defect question itself is supplied so that a forbidden unexpected event is not mistakenly declared irrelevant.

C01 [C] Does the person perform the primary interaction on the wrong object?
  Applicability code: target_object_specified
  Pre-video condition: The action specification or conditioning input identifies a target object.
C02 [C] Does the required primary interaction action fail to start at all?
  Applicability code: always
  Pre-video condition: Every video with a target HOI action.
C03 [C] Does the person only approach or pose without reaching the action's defining interaction event?
  Applicability code: defining_event_available
  Pre-video condition: The action has a defining event such as grasping, wearing, opening, lifting, or placing.
C04 [C] Does the object fail to reach the final state required by the action specification?
  Applicability code: final_state_specified
  Pre-video condition: The action specification defines an observable final state.
C05 [C] Is the body part performing the primary action inconsistent with the action specification?
  Applicability code: body_part_specified
  Pre-video condition: The action explicitly or inherently requires a particular body part.
C06 [C] Is the left or right hand used for the action inconsistent with the action specification?
  Applicability code: handedness_specified
  Pre-video condition: The action specification explicitly states left or right hand.
C07 [C] Is the number of hands participating in the primary action inconsistent with the action specification?
  Applicability code: hand_count_specified
  Pre-video condition: The action explicitly or inherently requires one hand or two hands.
C08 [C] Do the required sub-actions occur in the wrong order?
  Applicability code: ordered_multi_step_action
  Pre-video condition: The action contains two or more sub-actions with a required order.
C09 [C] Is the object's motion direction opposite to the direction specified by the action?
  Applicability code: direction_specified
  Pre-video condition: The action specifies a direction such as up, down, left, right, clockwise, or counterclockwise.
C10 [C] Is a required sustained action state lost before the specified phase ends?
  Applicability code: sustained_state_required
  Pre-video condition: The action requires holding, wearing, supporting, or maintaining an object state for a period.
S01 [S] During a phase that requires contact, does the specified body part never contact the target object?
  Applicability code: contact_required
  Pre-video condition: The action contains a phase that requires body-object contact.
S02 [S] While contact must be maintained, is there a clear air gap between the hand and the object surface?
  Applicability code: sustained_hand_contact
  Pre-video condition: The hand must remain in contact during a key phase and the contact region is visible.
S03 [S] When hand or body support is required, does the object float without any visible support?
  Applicability code: gravity_support_required
  Pre-video condition: The object should not self-float and the phase requires a hand, body part, or visible surface to oppose gravity.
S04 [S] Does the object translate or rotate as if controlled by the hand while clearly detached from it?
  Applicability code: hand_controlled_motion_expected
  Pre-video condition: The object's motion should be caused by the hand grasping, pushing, pulling, or supporting it.
S05 [S] Does required contact break unexpectedly before the action is complete?
  Applicability code: contact_until_completion_required
  Pre-video condition: The action requires continuous contact from contact establishment until completion.
S06 [S] Does the hand or a finger visibly penetrate into the solid interior of the target object?
  Applicability code: hand_object_boundary_visible
  Pre-video condition: The hand-object boundary is visible at one or more key moments.
S07 [S] Does the target object visibly penetrate an unintended body region?
  Applicability code: object_near_body
  Pre-video condition: The target object approaches the torso, arm, leg, or another body region not intended for contact.
S08 [S] Do the hand and object fuse into an inseparable single structure at the contact region?
  Applicability code: contact_boundary_visible
  Pre-video condition: The hand-object contact boundary is visible at one or more key moments.
S09 [S] Is the actual contact located on an object region unsuitable for completing the specified action?
  Applicability code: functional_contact_region_exists
  Pre-video condition: The action requires contact with a particular functional part or manipulable region.
S10 [S] Is the current grasp or support contact insufficient to oppose the object's gravity?
  Applicability code: object_supported_against_gravity
  Pre-video condition: The person is grasping, supporting, or suspending an object affected by gravity.
S11 [S] Is the finger-contact distribution incapable of forming the grasp type required by the action?
  Applicability code: finger_grasp_required
  Pre-video condition: The action requires visible fingers to grasp, pinch, clamp, or wrap around the object.
S12 [S] After placement is complete, does the object fail to contact the intended supporting surface?
  Applicability code: placement_on_surface
  Pre-video condition: The action requires placing the object on a table, floor, container, or another support surface.
S13 [S] Is the front-back occlusion order at the hand-object boundary inconsistent with their spatial positions?
  Applicability code: partial_hand_object_occlusion
  Pre-video condition: The hand and object partially occlude one another during a key phase.
S14 [S] During continuous contact, does the contact point jump to another object location without a geometric cause?
  Applicability code: continuous_contact_visible
  Pre-video condition: Contact is maintained and visible across consecutive frames.
S15 [S] Does the target object undergo an obvious solid collision with an unintended body part?
  Applicability code: object_approaches_unintended_body_part
  Pre-video condition: The target object's trajectory approaches a body part not intended for interaction.
S16 [S] After release, does the object remain suspended without a new support?
  Applicability code: release_without_new_support
  Pre-video condition: The person releases the object and it is not caught by another hand, body part, or surface.
M01 [M] Does the object begin controlled motion before control contact is established?
  Applicability code: contact_then_manipulation
  Pre-video condition: The hand should establish contact before pushing, pulling, grasping, or lifting the object.
M02 [M] After contact breaks, does the object continue copying the hand's controlled motion?
  Applicability code: visible_contact_break_during_motion
  Pre-video condition: A clear hand-object contact break is observable during motion.
M03 [M] During a stable hold, does the object's translation visibly lag behind the holding hand?
  Applicability code: stable_hold_with_translation
  Pre-video condition: The hand stably holds the object while visible translation occurs.
M04 [M] During a stable hold, does the object move without corresponding motion of the holding hand?
  Applicability code: stable_hold_visible
  Pre-video condition: The hand and object are in a stable hold and both motions are observable.
M05 [M] During a stable grasp, does the object slip unexpectedly relative to the palm?
  Applicability code: stable_grasp_without_slip_action
  Pre-video condition: The action requires a stable grasp and does not call for deliberate sliding or regrasping.
M06 [M] Does the object jitter independently without the motion propagating to the contacting hand?
  Applicability code: contact_during_visible_motion
  Pre-video condition: The hand and object remain in contact and temporal resolution is sufficient to observe short motions.
M07 [M] Does the target object's position teleport between adjacent moments in a way its velocity cannot explain?
  Applicability code: object_track_observable
  Pre-video condition: The target object is identifiable before and after the jump, or its track can be reliably matched across a brief occlusion.
M08 [M] Does the target object's orientation flip discontinuously between adjacent moments without a cause?
  Applicability code: object_orientation_observable
  Pre-video condition: The object's appearance or geometry supports tracking its orientation continuously.
M09 [M] Does new hand-object contact appear suddenly without the hand approaching the object?
  Applicability code: contact_establishment_visible
  Pre-video condition: The video contains a visible transition from separation to contact.
M10 [M] After release, is the object's gravity response implausibly delayed?
  Applicability code: unsupported_release_with_observable_aftermath
  Pre-video condition: The object has no new support after release and enough post-release video is available.
M11 [M] Does the interaction state suddenly reset to an earlier state without a cause?
  Applicability code: multi_phase_action
  Pre-video condition: The action progresses through two or more temporally distinguishable states.
M12 [M] After a brief occlusion, does the hand-object relation jump to a state incompatible with the pre-occlusion motion?
  Applicability code: brief_occlusion_with_track_recovery
  Pre-video condition: The hand-object relation undergoes a brief occlusion and can be reliably identified before and after it.
A01 [A] Does a visible interacting hand have a clearly incorrect number of fingers?
  Applicability code: interacting_fingers_countable
  Pre-video condition: The interacting fingers are clear enough at one or more key moments to assess their count.
A02 [A] Do two or more fingers merge non-anatomically into one structure?
  Applicability code: interacting_fingers_visible
  Pre-video condition: Finger contours of the interacting hand are visible at one or more key moments.
A03 [A] Does an interacting finger joint bend in a biomechanically impossible direction?
  Applicability code: finger_joint_pose_visible
  Pre-video condition: At least one interacting finger joint pose is sufficiently visible.
A04 [A] Does an interacting wrist disconnect from its corresponding forearm?
  Applicability code: wrist_forearm_region_visible
  Pre-video condition: The wrist-forearm connection on the interacting side is visible at one or more key moments.
A05 [A] Does an extra hand appear from nowhere during the interaction?
  Applicability code: upper_body_hands_observable
  Pre-video condition: Enough of the upper body is visible to distinguish the person's existing two hands.
A06 [A] Does an interacting hand disappear when it should be visible and is not occluded?
  Applicability code: interacting_hand_expected_visible
  Pre-video condition: Based on its trajectory, the interacting hand should be in frame and is not occluded by an object or the body.
A07 [A] Does the interacting arm length change abruptly in a way perspective cannot explain?
  Applicability code: interacting_arm_track_visible
  Pre-video condition: Enough shoulder, elbow, and wrist landmarks on the interacting side are visible over time.
A08 [A] Does the interacting hand collapse into an unrecognizable structure?
  Applicability code: interacting_hand_visible
  Pre-video condition: The interacting hand is visible at one or more key moments.
A09 [A] Does the target object disappear without being fully occluded or leaving the frame?
  Applicability code: target_object_expected_visible
  Pre-video condition: Based on its trajectory, the target object should remain in frame and is not fully occluded.
A10 [A] Does a second copy of the target object appear without cause during the interaction?
  Applicability code: single_target_object_expected
  Pre-video condition: The input and action specification contain only one target object.
A11 [A] Does a rigid target object's geometry deform without the action requiring deformation?
  Applicability code: rigid_object_or_shape_should_remain_stable
  Pre-video condition: The target is rigid, or the current action should not change its global shape.
A12 [A] Does the target object's scale jump between adjacent moments in a way depth change cannot explain?
  Applicability code: object_scale_trackable
  Pre-video condition: The target object is trackable over time and camera or object depth change can be assessed.

Return action_profile plus all 50 applicability judgments.
Complete instantiated judging request — exact example
SYSTEM
You are a strict, evidence-first judge of one atomic human-object-interaction defect question.
The question's semantic applicability was frozen before the generated video was viewed. Therefore NOT_APPLICABLE is forbidden in this request.

Inspect the complete video in temporal order, the timestamped overview, and every supplied high-detail frame strip. The target-object reference image is identity/part context only and is never evidence that generated contact or motion is correct.

State rules:
- DEFECT: visible generated-video evidence satisfies the supplied FAIL condition.
- CLEAN: positive inspection of the relevant phase establishes that the FAIL condition does not occur. Mere presence of a person/object, intended action, proximity, or an unverified tiny image is not enough.
- UNCERTAIN: the question is applicable, but crop, blur, occlusion, sampling, or conflicting evidence prevents a reliable decision. Uncertainty receives zero credit and must not be converted to CLEAN.

For hand-object contact, CLEAN requires inspecting actual boundaries: nearby is not contact, and a visible air gap is a defect when contact/support is required. For temporal questions, inspect motion rather than inferring it from a single frame. Cite real timestamps and F01-F17 frame IDs. Reasons must describe visible pixels/motion, not restate the prompt. Return exactly one judgment for the supplied QID.

USER
Evaluate exactly one applicable defect question.

Original requested prompt:
The young woman with light brown hair pulled back stands in a brightly lit room with white walls, holding a small, light green dropper bottle labeled HOIVG-Bench. The product is a small-size bottle, compact enough to be comfortably held in one hand. She gestures expressively with her hands while demonstrating and speaking directly to the camera, showing how to use the bottle and engaging the viewer with her presentation.

Object label: bottle
Frozen pre-video action profile:
{"actor_body_parts": ["hand"], "defining_event": null, "direction": null, "expected_object_count": 1, "functional_relation": "hand grasps the bottle to support it against gravity while gesturing", "gravity_support_required": true, "handedness": "unspecified", "main_action": "holding and demonstrating", "ordered_subactions": [], "release_or_placement_expected": false, "required_final_state": null, "required_hand_count": 1, "rigid_object_expected": true, "sustained_contact_required": true, "target_object": "bottle"}

Frozen applicability reason: The target object is specified as a bottle, so interacting with the wrong object is relevant. / 指定了目标物体瓶子,需检查是否对错误物体执行交互。

Question ID: C01
Axis: C
Defect question: Does the person perform the primary interaction on the wrong object?
缺陷问题: 人物是否把主要交互动作施加在了错误物体上?
FAIL condition: The primary interaction object is not the specified target, or the person operates only another object.
FAIL 条件: 主要交互对象不是指定目标物体,或人物只操作了另一个物体。
Critical: true

Fixed frame/timestamp mapping:
F01=0.000s
F02=0.200s
F03=0.400s
F04=0.600s
F05=0.800s
F06=1.000s
F07=1.200s
F08=1.400s
F09=1.600s
F10=1.760s
F11=1.960s
F12=2.160s
F13=2.360s
F14=2.560s
F15=2.760s
F16=2.960s
F17=3.160s

CLEAN must cite inspected evidence. DEFECT/UNCERTAIN must cite the narrowest useful worst interval. Use VIDEO in evidence_frame_ids when evidence falls between the 17 fixed frames. Return only this QID.
Structured output schema — exact C01 example
{
  "type": "object",
  "additionalProperties": false,
  "required": [
    "judgments"
  ],
  "properties": {
    "judgments": {
      "type": "array",
      "minItems": 1,
      "maxItems": 1,
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": [
          "question_id",
          "state",
          "evidence_timestamps",
          "evidence_frame_ids",
          "reason_zh",
          "reason_en",
          "confidence"
        ],
        "properties": {
          "question_id": {
            "type": "string",
            "enum": [
              "C01"
            ]
          },
          "state": {
            "type": "string",
            "enum": [
              "DEFECT",
              "CLEAN",
              "UNCERTAIN"
            ]
          },
          "evidence_timestamps": {
            "type": "array",
            "minItems": 1,
            "maxItems": 8,
            "items": {
              "type": "number",
              "minimum": 0
            }
          },
          "evidence_frame_ids": {
            "type": "array",
            "minItems": 1,
            "maxItems": 18,
            "items": {
              "type": "string",
              "enum": [
                "F01",
                "F02",
                "F03",
                "F04",
                "F05",
                "F06",
                "F07",
                "F08",
                "F09",
                "F10",
                "F11",
                "F12",
                "F13",
                "F14",
                "F15",
                "F16",
                "F17",
                "VIDEO"
              ]
            }
          },
          "reason_zh": {
            "type": "string"
          },
          "reason_en": {
            "type": "string"
          },
          "confidence": {
            "type": "number",
            "minimum": 0,
            "maximum": 1
          }
        }
      }
    }
  }
}

The runtime validates the single returned QID, allowed states, required English/Chinese rationales, timestamps within the clip, evidence IDs F01–F17 or VIDEO, and confidence in [0, 1]. Timestamp metadata can be supplemented from the fixed frame-ID mapping. Confidence is recorded for auditability and does not weight the score or represent calibrated human agreement.

Download the complete judging example, pre-video profile prompt example, profile response schema, and judging response schema.

5. Complete 50-question bank

All English question texts, applicability conditions, and FAIL conditions below are reproduced directly from the protocol configuration. Every question asks whether a defect is present. The critical marker is the stored question flag; cap triggers are specified separately in Section 2.

C · Action correctness and completion (10 questions)

IDDefect questionApplies whenFAIL condition
C01criticalDoes the person perform the primary interaction on the wrong object?The action specification or conditioning input identifies a target object.The primary interaction object is not the specified target, or the person operates only another object.
C02criticalDoes the required primary interaction action fail to start at all?Every video with a target HOI action.The person makes no recognizable attempt to perform the action on the target object.
C03criticalDoes the person only approach or pose without reaching the action's defining interaction event?The action has a defining event such as grasping, wearing, opening, lifting, or placing.The defining interaction event required by the action specification never occurs.
C04criticalDoes the object fail to reach the final state required by the action specification?The action specification defines an observable final state.Before the video ends, the target object does not reach the required final position, pose, or functional state.
C05Is the body part performing the primary action inconsistent with the action specification?The action explicitly or inherently requires a particular body part.The primary action is performed by the wrong body part.
C06Is the left or right hand used for the action inconsistent with the action specification?The action specification explicitly states left or right hand.The action is performed with the opposite hand from the specified side.
C07Is the number of hands participating in the primary action inconsistent with the action specification?The action explicitly or inherently requires one hand or two hands.The action requires one hand but uses two, or requires two hands but only one participates.
C08criticalDo the required sub-actions occur in the wrong order?The action contains two or more sub-actions with a required order.At least one required sub-action occurs before its prerequisite, invalidating the action logic.
C09Is the object's motion direction opposite to the direction specified by the action?The action specifies a direction such as up, down, left, right, clockwise, or counterclockwise.The primary object motion is clearly opposite to the specified direction.
C10criticalIs a required sustained action state lost before the specified phase ends?The action requires holding, wearing, supporting, or maintaining an object state for a period.The required state clearly ends before the specified phase is over and is not plausibly restored.

S · Contact, support, and spatial relations (16 questions)

IDDefect questionApplies whenFAIL condition
S01criticalDuring a phase that requires contact, does the specified body part never contact the target object?The action contains a phase that requires body-object contact.Throughout the key phase, the specified body part remains visibly separated from the target object.
S02criticalWhile contact must be maintained, is there a clear air gap between the hand and the object surface?The hand must remain in contact during a key phase and the contact region is visible.At least one clear frame shows a visible nonzero gap where the hand should contact the object surface.
S03criticalWhen hand or body support is required, does the object float without any visible support?The object should not self-float and the phase requires a hand, body part, or visible surface to oppose gravity.The object is clearly separated from every possible support yet remains suspended or stably stationary.
S04criticalDoes the object translate or rotate as if controlled by the hand while clearly detached from it?The object's motion should be caused by the hand grasping, pushing, pulling, or supporting it.A clear hand-object gap exists while the object continues to copy the hand's control trajectory.
S05criticalDoes required contact break unexpectedly before the action is complete?The action requires continuous contact from contact establishment until completion.Contact clearly breaks before the completion event even though the action does not require release.
S06criticalDoes the hand or a finger visibly penetrate into the solid interior of the target object?The hand-object boundary is visible at one or more key moments.The hand or finger crosses the outer surface into a rigid solid interior without the action requiring insertion.
S07criticalDoes the target object visibly penetrate an unintended body region?The target object approaches the torso, arm, leg, or another body region not intended for contact.The object surface crosses the body boundary into body volume without the action requiring it.
S08criticalDo the hand and object fuse into an inseparable single structure at the contact region?The hand-object contact boundary is visible at one or more key moments.The two-surface boundary becomes uninterpretable because hand texture or geometry continuously melts into the object.
S09Is the actual contact located on an object region unsuitable for completing the specified action?The action requires contact with a particular functional part or manipulable region.Contact occurs on a region that cannot produce the required functional effect.
S10criticalIs the current grasp or support contact insufficient to oppose the object's gravity?The person is grasping, supporting, or suspending an object affected by gravity.The visible contact lies where it cannot bear the load and no other support explains the object's stability.
S11Is the finger-contact distribution incapable of forming the grasp type required by the action?The action requires visible fingers to grasp, pinch, clamp, or wrap around the object.The visible fingers are not positioned to apply the required holding force.
S12criticalAfter placement is complete, does the object fail to contact the intended supporting surface?The action requires placing the object on a table, floor, container, or another support surface.In the completed placement state, a clear gap remains between the object and intended support surface.
S13Is the front-back occlusion order at the hand-object boundary inconsistent with their spatial positions?The hand and object partially occlude one another during a key phase.Within the same local relation, the foreground surface is covered by the background surface without motion that could explain the change.
S14During continuous contact, does the contact point jump to another object location without a geometric cause?Contact is maintained and visible across consecutive frames.The contact point changes discontinuously without sufficient relative hand-object motion to explain it.
S15criticalDoes the target object undergo an obvious solid collision with an unintended body part?The target object's trajectory approaches a body part not intended for interaction.The object hits an unintended body part without plausible avoidance, stopping, or collision response.
S16criticalAfter release, does the object remain suspended without a new support?The person releases the object and it is not caught by another hand, body part, or surface.Contact has clearly ended, yet the object remains fixed or suspended against gravity.

M · Hand-object motion synchronization and timing (12 questions)

IDDefect questionApplies whenFAIL condition
M01criticalDoes the object begin controlled motion before control contact is established?The hand should establish contact before pushing, pulling, grasping, or lifting the object.The object starts moving along the hand's subsequent trajectory before the hand contacts it.
M02criticalAfter contact breaks, does the object continue copying the hand's controlled motion?A clear hand-object contact break is observable during motion.After separation, the object continues on an approximately rigidly synchronized trajectory with the hand.
M03criticalDuring a stable hold, does the object's translation visibly lag behind the holding hand?The hand stably holds the object while visible translation occurs.The holding hand translates first and the object catches up later, creating a visible temporal delay.
M04criticalDuring a stable hold, does the object move without corresponding motion of the holding hand?The hand and object are in a stable hold and both motions are observable.The object undergoes clear rigid motion while the holding hand remains essentially stationary.
M05During a stable grasp, does the object slip unexpectedly relative to the palm?The action requires a stable grasp and does not call for deliberate sliding or regrasping.The object moves relative to the palm or fixed fingers without a regrasping action.
M06Does the object jitter independently without the motion propagating to the contacting hand?The hand and object remain in contact and temporal resolution is sufficient to observe short motions.The object shows one or more high-frequency position jumps without corresponding motion in the contacting hand.
M07criticalDoes the target object's position teleport between adjacent moments in a way its velocity cannot explain?The target object is identifiable before and after the jump, or its track can be reliably matched across a brief occlusion.The object appears at a clearly different position without traversing an intermediate path.
M08Does the target object's orientation flip discontinuously between adjacent moments without a cause?The object's appearance or geometry supports tracking its orientation continuously.The orientation changes substantially without enough hand or object motion to cause the rotation.
M09Does new hand-object contact appear suddenly without the hand approaching the object?The video contains a visible transition from separation to contact.Adjacent moments change from clear separation directly to contact without sufficient approach motion.
M10criticalAfter release, is the object's gravity response implausibly delayed?The object has no new support after release and enough post-release video is available.Contact has ended, but the object does not begin falling or responding until multiple discernible time steps later.
M11Does the interaction state suddenly reset to an earlier state without a cause?The action progresses through two or more temporally distinguishable states.The person, hand, or object abruptly returns to a previously completed state without a reversing action.
M12criticalAfter a brief occlusion, does the hand-object relation jump to a state incompatible with the pre-occlusion motion?The hand-object relation undergoes a brief occlusion and can be reliably identified before and after it.After occlusion, contact side, relative position, or grasp state changes in a way the pre-occlusion trajectory cannot explain.

A · Hand, limb, and object structure (12 questions)

IDDefect questionApplies whenFAIL condition
A01Does a visible interacting hand have a clearly incorrect number of fingers?The interacting fingers are clear enough at one or more key moments to assess their count.One hand visibly has fewer or more than five fingers, and occlusion does not explain the count.
A02Do two or more fingers merge non-anatomically into one structure?Finger contours of the interacting hand are visible at one or more key moments.Fingers that should be separate lack an explainable boundary and form one continuous finger-like structure.
A03criticalDoes an interacting finger joint bend in a biomechanically impossible direction?At least one interacting finger joint pose is sufficiently visible.A finger joint shows clear backward folding, out-of-range twisting, or an impossible joint connection.
A04criticalDoes an interacting wrist disconnect from its corresponding forearm?The wrist-forearm connection on the interacting side is visible at one or more key moments.A clear break, floating connection, or incorrect topology appears between wrist and forearm.
A05criticalDoes an extra hand appear from nowhere during the interaction?Enough of the upper body is visible to distinguish the person's existing two hands.A third hand appears simultaneously or sequentially without correspondence to either normal arm.
A06criticalDoes an interacting hand disappear when it should be visible and is not occluded?Based on its trajectory, the interacting hand should be in frame and is not occluded by an object or the body.The interacting hand is entirely absent in an unoccluded frame and later reappears without cause.
A07criticalDoes the interacting arm length change abruptly in a way perspective cannot explain?Enough shoulder, elbow, and wrist landmarks on the interacting side are visible over time.The arm visibly lengthens or shortens to reach the object without perspective or joint motion explaining it.
A08criticalDoes the interacting hand collapse into an unrecognizable structure?The interacting hand is visible at one or more key moments.Palm and finger topology is largely lost, melted, or replaced by an uninterpretable blob.
A09criticalDoes the target object disappear without being fully occluded or leaving the frame?Based on its trajectory, the target object should remain in frame and is not fully occluded.The target object is entirely absent when it should be visible and may later reappear without cause.
A10criticalDoes a second copy of the target object appear without cause during the interaction?The input and action specification contain only one target object.Two target copies appear simultaneously or sequentially without occlusion, reflection, or a pre-existing scene object explaining them.
A11Does a rigid target object's geometry deform without the action requiring deformation?The target is rigid, or the current action should not change its global shape.The object's contour or part proportions bend, melt, or stretch in a way viewpoint change cannot explain.
A12Does the target object's scale jump between adjacent moments in a way depth change cannot explain?The target object is trackable over time and camera or object depth change can be assessed.The object's image scale suddenly grows or shrinks without corresponding depth or camera motion.

Download the original bilingual JSON or English Markdown question bank.

6. Protocol downloads

The documentation download contains the complete question configuration, applicability and failure rules, category weights and caps, exact prompt examples, response schemas, and judge settings.

DownloadContents
Complete protocol documentation (ZIP)Questions, applicability and FAIL rules, prompts, response schemas, settings and checksums.
Original bilingual protocol JSONAll 50 questions, category weights and seven critical-failure rules.
English question bankExact question text, applicability conditions and FAIL conditions.
Complete judging promptExact system prompt and instantiated user request.
Pre-video profile promptComplete applicability system/user example with all 50 questions.
Profile response schemaAction profile and applicability response format.
Judging response schemaExact structured-output example for an applicable question.
Judge settingsRecorded model/backend, decoding settings and requested full-video sampling rate.

Sampling pseudocode

indices = [round(j * (N - 1) / 16) for j in range(17)]
timestamps = [index / fps for index in indices]

Scoring checks

If all applicable questions are CLEAN, the score is 100. If S02 alone is DEFECT, the contact category is capped at 50 and the overall score at 60. A C01, C02, or C03 DEFECT caps action at zero and therefore yields a zero harmonic overall score. The general formulas and all trigger rules are specified in Section 2.