The concept
Multimodal AI: a model that can take in more than one kind of data, such as text, images, audio, video or documents, in the same conversation.
So far in this series, models read text as tokens (Day 16). A multimodal model also cuts an image into small patches and audio into short slices, and turns each into numbers. Everything lands in the same "space" (embeddings, Day 11), so the model can reason across all of it at once.
Like a claims handler who reads the email, looks at the photos and listens to the voicemail before writing a single line.
Walkthrough: a dented bumper claim
Inputs from the driver's app:
- Photo: rear bumper, a dent about the size of a fist, paint scraped.
- Voice note: "I was reversing out of the grocery store lot around 6 pm yesterday and hit a concrete post. No one else involved, no injuries."
What the model does:
1. Looks at the image: rear bumper, visible dent, scratched paint, no broken lights.
2. Transcribes the audio into text (speech-to-text).
3. Pulls the facts out: date, time, location, single vehicle, no injuries, no third party.
4. Drafts the first notice of loss (the report that opens a claim): what happened, damage seen, parties involved, and fields it could not fill (e.g. "exact address not given").
5. Returns it as structured fields (Day 26) so the claims system can open the file.
The limits (important)
- A photo shows the surface. It can't see a bent bracket behind the bumper.
- The model can misread lighting, angles or reflections, and it can hallucinate (Day 22) details that aren't there.
- It should not decide coverage or pay the claim. A human adjuster verifies the photo, the story and the policy.
The win is speed and completeness at intake, not replacing the adjuster.
Where you see it at work
- Insurance: photo-based damage intake for auto and home claims.
- Healthcare: summarizing a scanned referral letter plus a clinician's dictated note, always with clinical review.
- Banking: reading a photographed cheque or ID document during onboarding.
- Retail: "snap a photo, find similar products" search.
- Business analysis: turning a workshop whiteboard photo into a list of requirements.
In one sentence
Multimodal AI reads images, audio and text together, which makes intake faster and richer, but a human still checks what the model thinks it sees.