---
title: How AI Decides What Counts as a Highlight
description: "Crowd noise, scoreboards, player tracking and video models: a plain-English guide to the signals AI uses to find sports highlights, and where they fall short."
image: https://useslipstream.ai/hubfs/05-featured-v3.png
---

[Skip to content](https://useslipstream.ai/blog/how-ai-decides-what-counts-as-a-highlight#main-content)

[![Slipstream Live Streaming](https://useslipstream.ai/hs-fs/hubfs/slipstream-bar-logo-v2.png?width=1457&height=416&name=slipstream-bar-logo-v2.png)Homepage](https://beta.strongholdlabs.io)

- [Product](https://useslipstream.ai/product)
- Solutions
  
    - [All solutions](https://useslipstream.ai/solutions)
    - [Event organizers](https://useslipstream.ai/solutions/event-organizers)
    - [League & business leaders](https://useslipstream.ai/solutions/league-leaders)
    - [Social & content teams](https://useslipstream.ai/solutions/social-content-teams)
    - [Teams & programs](https://useslipstream.ai/solutions/teams-programs)
    - [Broadcast & production](https://useslipstream.ai/solutions/broadcast-production)
    - [Platforms & partners](https://useslipstream.ai/solutions/platforms-partners)
    - [Athletes](https://useslipstream.ai/solutions/athletes)
- [Customers](https://useslipstream.ai/customers)
- [Technology](https://useslipstream.ai/technology)
- [Integrations](https://useslipstream.ai/integrations)
- [Events](https://useslipstream.ai/events)
- [Blog](https://useslipstream.ai/blog)

Get started

- [Product](https://useslipstream.ai/product)
- Solutions
  
    - [All solutions](https://useslipstream.ai/solutions)
    - [Event organizers](https://useslipstream.ai/solutions/event-organizers)
    - [League & business leaders](https://useslipstream.ai/solutions/league-leaders)
    - [Social & content teams](https://useslipstream.ai/solutions/social-content-teams)
    - [Teams & programs](https://useslipstream.ai/solutions/teams-programs)
    - [Broadcast & production](https://useslipstream.ai/solutions/broadcast-production)
    - [Platforms & partners](https://useslipstream.ai/solutions/platforms-partners)
    - [Athletes](https://useslipstream.ai/solutions/athletes)
- [Customers](https://useslipstream.ai/customers)
- [Technology](https://useslipstream.ai/technology)
- [Integrations](https://useslipstream.ai/integrations)
- [Events](https://useslipstream.ai/events)
- [Blog](https://useslipstream.ai/blog)

Get started

![How AI picks the highlight](https://useslipstream.ai/hs-fs/hubfs/05-featured-v3.png?width=1200&height=630&name=05-featured-v3.png)

# How AI Decides What Counts as a Highlight

![Chris](https://7528315.fs1.hubspotusercontent-na1.net/hub/7528315/hubfs/raw_assets/public/mV0_d-cms-elevate-theme_hubspot/elevate/images/avatar-placeholder.jpg?width=48&height=48&name=avatar-placeholder.jpg)

 Chris

November 5, 2026

AI for video

Ask ten producers what the best moment of a game was and you will get a few different answers. So how does a piece of software, watching hours of live video, decide which ten seconds deserve to become a clip? It does not have taste. It has signals. Here is what those signals are, in plain English, and where they still fall short.

## A highlight is a judgment call

A human editor uses everything at once. They see the ball go in, hear the crowd erupt, glance at the score, and know the context: this goal tied the match in the final minute. AI highlight systems try to rebuild that same instinct from separate pieces of evidence, then add them up into something like a "highlight score" for each moment.

Researchers have been doing this for years. One of the clearest early examples came from IBM Research, whose team built a system for the 2017 Masters that ranked golf moments using player reactions, crowd cheers and commentator excitement. Their [paper on arXiv](https://arxiv.org/abs/1707.07075) describes processing four live channels over four tournament days. That same mix of signals still shows up in most approaches today.

Five signals that feed a highlight score

1**What the camera sees** Actions and events: a goal, a card, a celebration.

2**What the venue sounds like** Crowd noise and commentator excitement.

3**What the scoreboard says** On-screen graphics read by OCR, plus live game data.

4**Where the players are** Tracking who is on screen and where they are on the field.

5**What a multimodal model understands** Models that watch, listen and describe the scene in words.

= Highlight score: keep, rank or skip

## The five signals, explained

**1. What the camera sees.** The most direct signal is spotting the event itself. In research this is called "action spotting": finding the exact second something happens. The [original SoccerNet dataset](https://arxiv.org/abs/1804.04527) helped kick this off with 500 full soccer games, about 764 hours, labeled for goals, cards and substitutions. The [SoccerNet 2024 challenges](https://arxiv.org/abs/2409.10587) pushed further, to pinpointing ball actions and even judging fouls from multiple camera angles.

**2. What the venue sounds like.** Crowds and commentators are free, built-in excitement meters. A 2025 study on professional soccer, [Della Santa and Lalli](https://arxiv.org/abs/2501.16100), trained one model on audio and one on video frames. The audio model reached 89% accuracy and the video model 83%, and combining them made the system more robust against false alarms and misses. Audio has a catch, though: the authors note it struggles when crowd noise stays loud for a long stretch.

**3. What the scoreboard says.** Sometimes the simplest clue is the most reliable. A [cricket highlights paper from a CVPR 2018 workshop](https://openaccess.thecvf.com/content_cvpr_2018_workshops/papers/w34/Shukla_Automatic_Cricket_Highlight_CVPR_2018_paper.pdf) read the on-screen score with OCR (software that reads text in images). If the run total jumped by four between shots, that was a boundary. A change in the wickets digit meant a wicket. Off-the-shelf tools do this reading today: Google's [Video Intelligence text detection](https://docs.cloud.google.com/video-intelligence/docs/text-detection) returns the text it finds along with when it appears on screen. Where an official data feed exists, it can play the same role without reading pixels at all.

**4. Where the players are.** Knowing who did something matters as much as knowing that it happened. The [SoccerNet Game State Reconstruction](https://arxiv.org/abs/2404.11335) work tackles this by turning broadcast video into a top-down minimap, identifying each player by jersey number, team and role. That is what lets a system tag a clip to a specific athlete, and it is also, by the authors' own description, a hard and still-open problem.

**5. What a multimodal model understands.** The newest layer is large AI models that take in video and audio and can describe what is happening. In a July 2025 write-up on the [AWS Machine Learning Blog](https://aws.amazon.com/blogs/machine-learning/accenture-scales-video-analysis-with-amazon-nova-and-amazon-bedrock-agents/), Accenture describes a highlights tool where separate AI agents handle the visuals, transcribe the audio, assemble the clip and then review it, with soccer, Formula 1 and rugby as examples. The appeal is flexibility: you can describe the moment you want instead of training a new detector for it.

## Adding it up

No single signal is enough. A roar might be for a different hole. A scoreboard graphic can update after the play is over. The IBM golf team found that blending signals ranked moments better than any one signal alone, and human judges rated 92.68% of the system's top clips as legitimate highlights, according to [the full paper](https://ar5iv.labs.arxiv.org/html/1707.07075).

The highlight score is not magic. It is a weighted vote between the eyes, the ears and the scoreboard.

## Why the sport matters

A dunk, a wave and a wicket look nothing alike. A dunk is a burst of motion near the rim. A great wave in surfing can build for many seconds before the moment that scores. A wicket might be a tiny deflection that only makes sense once the umpire's finger goes up. A system tuned for one will misread another.

Even the IBM golf researchers flagged that one of their components, detecting golf's TV graphics to find where a shot starts, was specific to golf and would need rework elsewhere. And general-purpose AI still stumbles on fast, technical sport. In the 2025 [FineBadminton](https://arxiv.org/abs/2508.07554) benchmark, the best commercial model tested, Gemini 2.5 Pro, answered about 39% of multiple-choice questions correctly. A [7-billion-parameter open model fine-tuned on badminton data](https://arxiv.org/html/2508.07554) beat it, at about 42%. The lesson: sport-specific context and tuning matter.

## The honest limits

Every signal has blind spots worth knowing about:

1**Noise is not always news** Loud crowds can belong to a different moment, a problem both the golf and soccer papers above call out.

2**Models skim** Many video models look at sampled frames, not every frame. Amazon's [Nova documentation](https://docs.aws.amazon.com/nova/latest/userguide/modalities-video.html) describes sampling one frame per second for shorter videos. A fast edge or a flick of the wrist can fall between frames.

3**Camera angles matter** The 2025 soccer study notes its video model could be misled by camera position. Unusual angles mean weaker results.

4**Story still needs people** AI can find the moment. Knowing it is a comeback, a farewell or a first career win often takes someone who knows the athletes.

None of this means AI highlights are not ready. It means the best results come from combining signals, tuning for the sport and keeping a human in the loop for the moments that carry the story. As we argued in [The Camera Problem Is Solved](https://go.strongholdlabs.io/stronghold-labs-blog/the-camera-problem-is-solved.-the-content-problem-is-the-opportunity), capturing the video is no longer the hard part. Turning it into the right clips, fast, is.

## Where Slipstream fits

Slipstream, by Stronghold Labs, takes the live video your organization already produces and turns it into timely, finished, searchable, publishable content, with highlights delivered in under a minute. It works on top of your existing cameras and streams. Under the hood, the pipeline runs Ingest, Understand, Segment, Clip, Enrich, Transform, Organize and Publish. Through our platform partner LiveHeats, athletes also get a locker of their own clips.

[Book a demo](https://go.strongholdlabs.io/demo)

### Go deeper

- [Merler et al. (IBM Research): Automatic Curation of Golf Highlights using Multimodal Excitement Features](https://arxiv.org/abs/1707.07075)
- [Della Santa and Lalli: Automated Detection of Sport Highlights from Audio and Video Sources](https://arxiv.org/abs/2501.16100)
- [Shukla et al.: Automatic Cricket Highlight Generation Using Event-Driven and Excitement-Based Features](https://openaccess.thecvf.com/content_cvpr_2018_workshops/papers/w34/Shukla_Automatic_Cricket_Highlight_CVPR_2018_paper.pdf)
- [SoccerNet 2024 Challenges Results](https://arxiv.org/abs/2409.10587)
- [SoccerNet Game State Reconstruction: athlete tracking and identification on a minimap](https://arxiv.org/abs/2404.11335)
- [FineBadminton: fine-grained badminton video understanding benchmark](https://arxiv.org/abs/2508.07554)
- [Google Cloud: Video Intelligence text detection (OCR)](https://docs.cloud.google.com/video-intelligence/docs/text-detection)
- [Amazon Nova: video understanding documentation](https://docs.aws.amazon.com/nova/latest/userguide/modalities-video.html)
- [AWS Machine Learning Blog: Accenture scales video analysis with Amazon Nova and Amazon Bedrock Agents](https://aws.amazon.com/blogs/machine-learning/accenture-scales-video-analysis-with-amazon-nova-and-amazon-bedrock-agents/)

## Share this post

<https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fuseslipstream.ai%2Fblog%2Fhow-ai-decides-what-counts-as-a-highlight><https://twitter.com/intent/tweet?url=https%3A%2F%2Fuseslipstream.ai%2Fblog%2Fhow-ai-decides-what-counts-as-a-highlight><https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fuseslipstream.ai%2Fblog%2Fhow-ai-decides-what-counts-as-a-highlight><https://pinterest.com/pin/create/button/?url=https%3A%2F%2Fuseslipstream.ai%2Fblog%2Fhow-ai-decides-what-counts-as-a-highlight>[mailto:https%3A%2F%2Fuseslipstream.ai%2Fblog%2Fhow-ai-decides-what-counts-as-a-highlight](mailto:https%3A%2F%2Fuseslipstream.ai%2Fblog%2Fhow-ai-decides-what-counts-as-a-highlight)

## Keep reading

### [AI News This Week: Google Ships a Fast, Cheap AI Judge for Video (Nov 13)](https://useslipstream.ai/blog/ai-news-this-week-google-ships-a-fast-cheap-ai-judge-for-video-nov-13)

### [AI News This Week: Google's Gemini Omni Gets Longer, Cheaper Clips (Nov 06)](https://useslipstream.ai/blog/ai-news-this-week-googles-gemini-omni-gets-longer-cheaper-clips-nov-06)

[![Slipstream Live Streaming](https://useslipstream.ai/hs-fs/hubfs/slipstream-bar-logo-v2.png?width=1457&height=416&name=slipstream-bar-logo-v2.png "Slipstream Live Streaming")](https://beta.strongholdlabs.io)

- [Product](https://useslipstream.ai/product)
- Solutions
- [Customers](https://useslipstream.ai/customers)
- [Technology](https://useslipstream.ai/technology)
- [Integrations](https://useslipstream.ai/integrations)
- [Events](https://useslipstream.ai/events)
- [Blog](https://useslipstream.ai/blog)

<https://www.linkedin.com><https://www.facebook.com><https://www.twitter.com><https://www.instagram.com><https://www.tiktok.com>

---

Privacy Policy · Legal · © 2026. All rights reserved.

```json
{
  "@context" : "https://schema.org",
  "@type" : "BlogPosting",
  "author" : {
    "@type" : "Person",
    "name" : "Chris",
    "url" : "https://useslipstream.ai/blog/author/chris"
  },
  "dateModified" : "2026-11-05T15:00:01.267Z",
  "datePublished" : "2026-11-05T15:00:01.000Z",
  "headline" : "How AI Decides What Counts as a Highlight",
  "image" : [ "https://useslipstream.ai/hubfs/05-featured-v3.png" ],
  "mainEntityOfPage" : {
    "@id" : "https://useslipstream.ai/blog/how-ai-decides-what-counts-as-a-highlight",
    "@type" : "WebPage"
  },
  "publisher" : {
    "@type" : "Organization",
    "logo" : {
      "@type" : "ImageObject",
      "url" : "https://useslipstream.ai/hubfs/slipstream-bar-logo-v2.png"
    }
  }
}
```