GuideSight.doc - WordPadケーススタディ
← Back to projects もどる
GuideSight

GuideSight

A harness sensor and phone app that warn guide-dog walkers about obstacles ahead, built at HackCamp 2026

Team

Donna Li

Jake Fung

Sam

My role

Hardware

Backend

Frontend

Tools

ESP32

HC-SR04

Flask

React

Gemini Vision

ElevenLabs

Duration

Hackathon · HackCamp 2026

Overview

GuideSight is an assistive walking companion for people who use guide dogs. A sensor on the harness and a phone app work together to give obstacle alerts and spoken guidance, combining hardware sensing with AI scene analysis.

Devpost submission video, HackCamp 2026.

Problem

A guide dog steers around obstacles, but it can't tell its handler what's ahead, how far, or how urgent it is. Curbs, drop-offs, and oncoming traffic all get the same silence.

No distance warning

A dog stops or swerves, but the handler gets no sense of how close an obstacle is until they're on top of it.

No hazard severity

A curb, a staircase, and a moving car all call for different responses, and none of that distinction reaches the handler.

No hands-free alert

A phone alone can't be checked mid-walk, and a cane-style buzzer alone can't describe a scene.

Hardware

An ESP32 and an ultrasonic sensor ride on the harness.

An HC-SR04 ultrasonic sensor measures distance to whatever is ahead, and an ESP32 microcontroller reads it and sends it over Wi-Fi to the backend. A buzzer and an optional vibration motor sit on the harness handle, so the beep itself tells the handler how close something is.

  • Distance thresholds trigger progressive warnings: the beep speeds up as the obstacle gets closer.
  • The ESP32 streams readings to the Flask backend over Wi-Fi rather than processing hazards on-device.
Wiring diagram for the ESP32, HC-SR04 ultrasonic sensor, buzzer, and vibration motor on the harness
Wearable wiring: ESP32, HC-SR04, buzzer, and vibration motor.
Diagram showing where the sensor and electronics mount on a guide-dog harness
Where the hardware mounts on the harness.

Software

A Flask backend with WebSocket support (flask-sock) takes the sensor stream and the phone's camera frames, backed by Firestore. The walker-facing app is React, Vite, and Tailwind. Google Maps' Places, Routes, and Geocoding APIs handle navigation alongside the obstacle alerts.

Architecture diagram: harness sensor, phone app, Flask backend, Firestore, Gemini Vision, and Google Maps APIs
System architecture: harness, phone app, backend, and the AI and maps services it calls.

Gemini Vision

The phone's camera becomes a second sensor.

Every 15 seconds, or every 6 seconds once the harness senses something nearby, the phone sends a camera frame to Gemini Vision. Its scene description feeds a hazard classifier on the server, which sorts what it sees into trip hazards, drop-offs, or traffic danger.

Obstacle warning system

The hazard tier decides how hard the system interrupts the walk:

  • Trip hazards, like curbs or potholes, get a spoken warning without stopping navigation.
  • Drop-offs and stairs always get an urgent response, with a stop screen inside one meter.
  • Dangers like red lights or moving traffic trigger an immediate stop screen regardless of distance.

Next steps

The harness sensor, Gemini Vision pipeline, and tiered alerts work end to end for the hackathon demo. Next would be testing the distance thresholds and hazard tiers with an actual guide-dog handler, and moving hazard classification closer to the harness to cut latency.

Thank you

To Jake and Sam, built together at HackCamp 2026.

Behind the scenes

Wiring the sensor into the harness pocket, testing the walking mode and the accessibility settings on the phone, the app running next to its code, and the three of us heads-down at HackCamp 2026.

Wiring the sensor and buzzer into the harness pocket.
Donna, Jake, and Sam working on laptops around a table at HackCamp 2026
Heads-down at HackCamp 2026.
Setting up the accessibility voice and vibration options.
An ESP32 dev board plugged in via USB, with an Arduino behind it on the table
The ESP32 plugged in and ready to flash.
Testing walking mode's obstacle narration.
The app running next to its own dev console.

Attachments てんぷ

Ready 準備完了9 sections