Introduction

Observing someone perform a task (e.g., cooking, assembling furniture or fixing an electronic device) is a common approach for humans to acquire new skills. Instructional videos provide an excellent resource to learn such procedural activities for both humans and AI agents. However, the problem with using instructional videos from the web is that they tend to be long and noisy, i.e., only a limited number of frames in the video correspond to the instruction steps, while the remaining video segments are unrelated to the task (e.g., title frames, close-ups of people talking, and product advertisements). Thus, a major challenge when dealing with instructional videos is filtering out the uninformative frames and focusing only on the task-relevant segments, i.e., the key-steps. As a result, many recent efforts tackle the problem of instruction key-step localization, e.g., [2, 3, 4, 5, 6, 7]. However, most previous work aimed at temporally localizing key-steps from instructional videos relies on some form of supervision, and are therefore not deployable at scale and without human supervision.

In this blog, we present StepFormer [1], a novel self-supervised approach that simultaneously discovers and temporally localizes procedure key-steps in long untrimmed videos without requiring human annotation nor step descriptions, as illustrated in Figure 1.

Figure 1.  StepFormer for instruction step discovery and localization. StepFormer is a transformer decoder trained to discover instruction steps in a video, supervised purely from video subtitles. At inference, it only needs the video to discover an ordered sequence of step slots and temporally localize them in the video.

StepFormer Training and Inference Procedures

StepFormer is our model for procedure step discovery in instructional videos. Given an N second long video as input, StepFormer returns K step slots, s – a sequence of vectors capturing ordered instruction steps in the video. We train StepFormer on a large dataset of instructional videos, HowTo100M [5], without any supervision, by temporally aligning step slots with the narrations that accompany the video. Our full training pipeline is summarized in Figure 2.

Figure 2.  StepFormer training overview. (left) We first embed an untrimmed instructional video with a frozen UniVL encoder [8]. Next, we attend to the video with our StepFormer with learned step queries to extract step slots. (right) To form the training targets, we take the corresponding video subtitles, extract a sequence of verb phrases, and embed them with the UniVL text encoder [8]. (middle) To supervise StepFormer, we find a matching subsequence between the step slots and verb phrases via seq-to-seq alignment with outlier rejection [2]; the green entries in the alignment matrix denote correspondences. The resulting alignment is used to define a contrastive loss.

StepFormer is implemented as a transformer decoder with learnable input queries. Similar to the learnable object queries in DETR [9], StepFormer’s queries learn to attend to informative video segments and thus can be viewed as step proposals. To enforce the output step slots to follow the temporal order, we use an order-aware loss based on temporal alignment of the learned steps and video narrations. Since video narrations tend to be noisy and do not always describe visually groundable steps, we use a flexible sequence-to-sequence alignment algorithm previously designed by our team, DropDTW [9], which allows for non-alignable narrations to be dropped. With this alignment technique, we obtain a set of paired step slots and narrations. We use these pairings to form positive and negative pairs and rely on contrastive learning as our main supervisory signal. In addition, we introduce two regularization techniques: (1) slot smoothness, which enforces that each slot corresponds to continuous set of frames in the video and (2) slot diversity, which enforces slots to represent different steps in the video.

Figure 3.  Step localization inference. Given a video, StepFormer first extracts a sequence of step slots. The step slots can be interpreted as temporally ordered key-step candidates. Next, it aligns the sequence of step slots with the video sequence, using Drop-DTW [1]. This step puts the elements of the two sequences in correspondence, while identifying outliers. Here, the step slots and video frames of the same color are matched by Drop-DTW, while the white slots and frames are dropped from the alignment. The colored video segments represent the final step localization result.

At inference time, StepFormer detects and localizes key-steps without requiring narrations. StepFormer’s inference procedure is illustrated in Figure 3. It takes an instructional video as input and returns an ordered sequence of K step slots conditioned on this video. Each step slot corresponds to a procedural key-segment in the video and carries its semantics. However, given that we use a large fixed number of K step slots for all videos, some step slots may be duplicates or have weak binding with the video segment. For this reason, we need to select a subset of the step slots that concisely describe the given video. For this, we once again rely on sequence-to-sequence alignment using Drop-DTW as our inference procedure for step localization.

In summary, our work proposes an efficient architecture that can capture the ordered set of key steps that happen in an instructional video without requiring any annotations for training and that only requires the video input at inference time.

Evaluation Results

Given that our approach does not need human annotation, we train it on the largest un-annotated instructional video dataset, HowTo100M [5]. Following previous work [10], we evaluate our proposed method on the CrossTask [7] and ProceL [11] datasets. We also report results on the largest annotated instructional video dataset, i.e., COIN [6]. For all evaluations, the same model pre-trained on HowTo100M is used directly, without finetuning or adaptation to the downstream. Our results speak decisively in favor of our approach where we outperform previous baselines with sizeable margins on all standard metrics. Overall, the obtained results (i) highlight StepFormer’s capability to automatically discover key-steps even while being completely unsupervised and having no access to the datasets used for evaluation during training; and (ii) also show that our method yields better localization, as highlighted in the qualitative results in Figure 4.

Figure 4.  Qualitative comparison of temporal localization. Comparison of our self-supervised StepFormer with the weakly supervised baselines on CrossTask.

Conclusion

In summary, this work has introduced an efficient architecture that can capture the ordered set of key steps that happen in an instructional video without requiring any annotations for training and that only requires the video input at inference time. Evaluation results on multiple benchmarks and metrics show the superiority of our approach.

Link to the paper