Building binAI: A Vertex AI Mobility Radar for the Visually Impaired

Navigating complex physical environments is a daily challenge for the visually impaired. While standard GPS apps help with macro-navigation, they fail to detect immediate physical obstacles, read environmental text, or recognize faces in real-time.

To bridge this gap, Muhammad Zubair engineered binAI—a cutting-edge, voice-controlled Android application that acts as a real-time mobility instructor and visual assistant.

By combining lightning-fast on-device sensors with powerful Google Vertex AI cloud models, binAI keeps users safe and aware of their surroundings with zero latency.

The Tech Stack & Architecture

Building a real-time mobility radar requires a highly optimized, concurrent architecture. I designed binAI using a hybrid Edge-to-Cloud approach to ensure maximum speed and reliability.

The Hybrid Edge-to-Cloud Pipeline

To provide the fastest and safest experience, binAI cannot rely solely on the cloud (which introduces latency) or solely on the phone (which lacks deep reasoning). Here is how the hybrid architecture solves this:

1. The Edge (Phone): The Android app uses Kotlin Coroutines to process camera frames locally at high speeds, detecting bounding boxes, calculating obstacle distances in feet, and finding faces.

2. The Cloud (Server): The app injects this mathematical sensor data into a prompt alongside a compressed image and sends it to the FastAPI backend.

3. The Brain (Vertex AI): The Gemini model reads the image and the exact sensor data to generate highly accurate, spatial, and urgent instructions (e.g., "STOP! Stairs going down").

Code Spotlight: Solving Edge-Case Complexities

1. Fixing Mobile Camera Rotation for Face Recognition

A major issue with Android cameras is that they often send frames rotated by 90 degrees, causing standard face recognition libraries to fail. I engineered a fallback mechanism using NumPy to rotate the image matrix and fix the underlying C++ memory contiguity before re-scanning.

# 1. Look for faces in the standard orientation
face_locations = face_recognition.face_locations(img)

# 2. THE ROTATION FIX (With C++ Memory Fix)
if not face_locations:
    # Rotate -90 degrees and fix memory layout for the C++ backend
    img_rotated = np.ascontiguousarray(np.rot90(img, k=-1))
    face_locations = face_recognition.face_locations(img_rotated)
    
    if face_locations:
        img = img_rotated
    else:
        # Fallback: Rotate +90 degrees
        img_rotated = np.ascontiguousarray(np.rot90(img, k=1))
        face_locations = face_recognition.face_locations(img_rotated)
        if face_locations:
            img = img_rotated

2. Sensor Data Injection into Vertex AI

To prevent the LLM from hallucinating distances, the backend dynamically injects the exact mathematical data calculated by the phone's edge sensors directly into the system prompt.

@app.post("/navigate")
async def navigate(image: UploadFile = File(...), on_device_data: Optional[str] = Form(None)):
    
    # Injecting Edge ML data into the Cloud LLM prompt
    sensor_injection = f"ON-DEVICE SENSOR DATA: {on_device_data}. Use these exact distances for your evasion commands." if on_device_data else ""
    
    prompt = f"""
    You are a real-time mobility radar for a blind person walking forward.
    {sensor_injection}
    
    CRITICAL CONSTRAINTS:
    - Do NOT use full sentences. 
    - Do NOT be polite. 
    - Prioritize distance (feet/steps) and directional commands (left/right).
    """
    
    result = call_vision_model(prompt, await image.read())
    return {"status": "success", "script": result}

The Impact

binAI demonstrates how modern AI infrastructure—when combined with thoughtful, accessibility-first mobile engineering—can directly improve human lives. By leveraging Google Cloud Run and Vertex AI, the application remains highly scalable, cost-effective, and incredibly fast.