FridgeAI documentation

Everything you need to use FridgeAI as a cook, and to understand or extend it as a developer. The app turns a photo of your ingredients into meals you can actually make.

Overview

FridgeAI answers one question — “what can I make with this?” — from a single photo. Its loop is Photo → Detect → Confirm → Choose → Cook: a vision model reads your ingredients, you confirm and edit them, optional preferences narrow things down, and it proposes recipes ranked by how much you already own. A hands-free Cook Mode and an in-context assistant carry you through the actual cooking.

It is mobile-first (you’ll usually photograph ingredients with your phone) and works fully offline with a built-in sample kitchen, so the demo is always walkable.

Getting in

The demo is gated so a shared link is safe — only the person holding the inbox gets in.

1. Open the app link and enter your email address.
2. A fresh one-time 6-digit PIN is generated. It’s emailed to you when email delivery is configured, and otherwise shown on screen for testing.
3. Enter the PIN and the live app opens. Closing the tab ends the session.

A day in the life

It’s 6:40pm. You open the fridge: some tomatoes, a carton of eggs, half a bag of spinach, an onion or two, a wedge of cheddar, a couple of carrots. Normally this is where you’d give up and order in.

Instead you open FridgeAI, tap Scan your fridge, and take one photo. A moment later — “Analyzing your fridge…” — chips appear: Tomatoes ×3, Eggs ×6, Spinach, Onion ×2, Possibly Cheddar. The cheddar was a guess, so you tap it to confirm; you also add the milk it couldn’t see. You tap Find meals, pick Vegetarian and Under 30 min, and hit Show me recipes.

Up comes a Spinach & Egg Skillet — 20 min, uses 6 of 7, high protein — sitting above a tomato soup and a yogurt bowl. A Use these first panel reminds you the spinach won’t keep long. You open the skillet, see you already have everything except a clove of garlic, tap Start cooking, and step through it one instruction at a time with a timer for the eggs. Halfway through you ask, “make it spicier” — it suggests the chilli flakes in your cupboard. Dinner, from what you already had, in about a minute of planning.

The five-step flow

1 · Upload

Upload an image, drag-and-drop, or take a photo (the camera opens the rear lens on phones). Or hit Try an example to walk the flow with a sample kitchen — no photo needed.

2 · Detect

The image is analysed and returns a structured ingredient list. Amounts are honest phrases (“about 3”, “a small bunch”), never invented precise counts, and low-confidence items are shown as “Possibly…”.

3 · Confirm

This step is deliberate — recipes never run on raw detection. Rename, adjust quantities, remove wrong items, and + Add ingredient for anything missed. Category dots and an “uncertain” style help you scan.

4 · Choose

Optional preference chips, then 3–6 recipes ranked by availability. Each card shows time, difficulty, calories, protein, how many of your ingredients it uses, and how many are missing.

5 · Cook

Open a recipe for the full detail — what you already have, optional additions, and numbered steps — then Start cooking for the step-by-step Cook Mode.

Preferences

All optional, all compact chips — no long form:

Cook Mode & the assistant

Cook Mode shows one instruction at a time in large type, with Previous / Next controls and a per-step timer when a step mentions a duration (“cook for 3 minutes” → a 3-minute timer). It’s built for a phone on the counter and one free hand.

Ask FridgeAI is a grounded assistant that already knows your selected recipe, your confirmed ingredients, your edits, and your preferences — so you never repeat context. Try:

• “I don’t have onion.” · “Can I replace eggs?”
• “Make this spicier.” · “Can I cook this in an air fryer?”
• “Make this vegetarian.” · “How much should I cook for four?”

Safety & honesty

Match scores are understandable, not mystical: “Uses 8 of your 9 ingredients” or “92% ready” — computed locally from your list, never a raw AI confidence number.

Architecture

The AI is a modular service layer, decoupled from the UI and from any single provider. Each stage validates its output against a schema before it reaches the interface, and falls back to a deterministic engine when the model is unavailable.

app/
  page.tsx              flow orchestrator (stage machine)
  api/detect|recipes|chat/route.ts   thin server handlers (key stays server-side)
services/ai/
  schemas.ts    zod contracts + inferred types (single source of truth)
  client.ts     provider transport: validated JSON + token streaming
  vision.ts     image  -> detected ingredients   (falls back to mock)
  recipes.ts    pantry -> recipes                 (falls back to mock)
  chat.ts       grounded assistant + safety guardrails (offline answers)
  mock.ts       deterministic detection + recipe generation
  shelfLife.ts  careful "use these first" perishable hints
lib/
  store.ts      flow state (reducer + sessionStorage)
  match.ts      local recipe <-> pantry matching & ranking

Provider precedence is gateway model → deterministic offline fallback. The application stays keyless; the key lives only in the gateway, which also redacts anything identifying from outbound prompts.

API reference

Three server routes, each POST, each returning validated data.

POST /api/detect

Image → structured ingredients.

// request
{ "imageDataUrl": "data:image/jpeg;base64,…" }

// response
{
  "ingredients": [
    { "name": "Tomatoes", "quantityEstimate": "about 3",
      "confidence": 0.94, "category": "vegetable" },
    { "name": "Cheddar", "quantityEstimate": "a wedge",
      "confidence": 0.68, "category": "dairy", "uncertain": true }
  ],
  "source": "model"        // or "mock" (offline fallback)
}

POST /api/recipes

Confirmed ingredients + preferences → ranked recipes.

// request
{ "ingredients": ["Tomatoes","Eggs","Spinach","Onion"],
  "preferences": { "diet":"vegetarian", "time":"under-30", "cuisine":"any" } }

// response
{ "recipes": [ /* Recipe[] — see Data models */ ], "source": "model" }

POST /api/chat

A grounded question → a streamed text/plain answer. The body carries the selected recipe, the user’s ingredients, and preferences, so the assistant needs no repetition.

// request
{ "question": "Can I replace eggs?",
  "recipe": { /* the selected Recipe */ },
  "ingredients": ["Eggs","Spinach","Onion"],
  "preferences": { "diet":"vegetarian", "time":"any", "cuisine":"any" } }

// response: streamed plain-text tokens

Data models

type Ingredient = {
  id: string
  name: string
  quantityEstimate?: string     // honest phrase, not a fake count
  category?: "vegetable" | "fruit" | "protein" | "dairy"
            | "grain" | "herb" | "condiment" | "pantry" | "other"
  confidence?: number           // 0..1
  uncertain?: boolean           // renders as "Possibly …"
  userConfirmed: boolean
}

type Recipe = {
  id: string
  title: string
  description: string
  cookingTimeMinutes: number
  difficulty: "easy" | "medium"
  calories: number
  proteinGrams: number
  cuisine?: string
  tags: string[]
  ingredients: { name: string; optional: boolean; note?: string }[]
  steps: string[]
}

Running & deploying

Local, from the app repository:

npm install
npm run dev          # http://localhost:3120

It runs fully with no key (deterministic offline mock). To enable live photo analysis and tailored recipes, set a vision-capable model key in .env.local; the key is read server-side only and never reaches the browser.