Application architecture

How a question travels from the search box to a set of recommendations, how the advisor index is built, and where each piece runs.

System overview

The browser talks to one service. That service owns the retrieval logic and is the only thing holding model credentials — the frontend never calls a model provider directly. Authentication is the exception: it goes straight from the browser to Supabase.

flowchart TB
    student([Student])

    subgraph browser["Browser"]
        direction TB
        pages["Astro static pages<br/>index · login · signup · dashboard"]
        islands["React islands<br/>SearchBar · CvUpload · AuthButtons"]
        pages --- islands
    end

    subgraph service["FastAPI service"]
        direction TB
        routes["api/routes.py<br/>validation · error mapping · NDJSON streaming"]
        subgraph svc["services/"]
            direction LR
            qexp["query_expansion"]
            embed["embeddings"]
            advisors["advisors"]
            cvsvc["cv"]
        end
        subgraph core["core/"]
            direction LR
            clients["clients<br/>lazy, provider-aware"]
            stream["streaming<br/>JSON scanner"]
            params["model_params<br/>capability fallback"]
        end
        routes --> svc
        svc --> core
    end

    subgraph vendors["External services"]
        direction TB
        provider["Azure OpenAI / OpenAI<br/>chat + embeddings"]
        supabase["Supabase<br/>authentication"]
    end

    qdrant[("Qdrant<br/>18,333 chunks<br/>1536-dim cosine")]

    student --> browser
    islands -- "POST /api/retrieve_advisors" --> routes
    islands -- "POST /api/match_cv_advisors" --> routes
    islands -. "sign in / sign up" .-> supabase
    core --> provider
    advisors --> qdrant

    classDef ext fill:#fef3c7,stroke:#f59e0b,color:#78350f
    classDef store fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a
    class provider,supabase ext
    class qdrant store

Amber marks third-party services; blue marks the vector store.

A search, step by step

A single search makes several round trips before anything is displayed. The relevance check runs first so an off-topic question is rejected before any embedding or search work is paid for.

sequenceDiagram
    autonumber
    actor S as Student
    participant UI as Browser
    participant API as FastAPI
    participant M as Model provider
    participant Q as Qdrant

    S->>UI: Types a research question
    UI->>API: POST /api/retrieve_advisors
    API->>M: Is this about research?
    M-->>API: yes / no
    Note over API: Off-topic stops here with 422

    API->>M: Rewrite into 4 search queries
    M-->>API: Subqueries

    loop For each subquery
        API->>M: Embed the query
        M-->>API: 1536-dim vector
        API->>Q: Cosine search, top 5
        Q-->>API: Scored chunks
    end

    Note over API: Merge, drop duplicates, keep top 10

    API->>M: Write cards from the shortlist
    loop As each card completes
        M-->>API: JSON fragment
        API-->>UI: One NDJSON line
        UI-->>S: Card appears
    end

Cards are streamed as they are written, so the first appears well before the last.

Collecting the data

Rankings on csrankings.org are rendered by JavaScript, so the ranking pages have to be driven through a headless browser rather than fetched. Individual faculty homepages are a different problem: most are plain HTML and are fetched directly, but the ones that need JavaScript fall back to the browser. The fast path is tried first because it is roughly an order of magnitude cheaper per page.

flowchart TB
    subgraph sources["Sources on csrankings.org"]
        direction LR
        rank["Ranking pages<br/>rendered by JavaScript"]
        csv1[("csrankings.csv<br/>name · homepage · scholar id")]
        csv2[("generated-author-info.csv<br/>name · dept · area · year · count")]
    end

    subgraph crawl["crawler/crawler.py"]
        direction TB
        sel["Headless Chrome<br/>one pass per CS area"]
        soup["BeautifulSoup<br/>parse #ranking table"]
        list["Universities and advisors"]
        seen{"Already visited?"}
        fast["requests.get<br/>5s timeout"]
        ok{"Usable HTML?"}
        fallback["Selenium fallback<br/>for JavaScript homepages"]
        err["Record error, keep going"]
        sel --> soup --> list --> seen
        seen -- yes --> skip["Skip"]
        seen -- no --> fast --> ok
        ok -- yes --> got["Profile text"]
        ok -- no --> fallback --> got
        fallback -. fails .-> err
    end

    subgraph merge["crawler/storage.py"]
        direction TB
        prev[("Previous crawl")]
        combine["Merge with previous run"]
        out[("data/raw/csrankings_TIMESTAMP.json")]
        prev --> combine --> out
    end

    parse["parse_scraped_data.py<br/>normalise · paper counts per area"]

    rank --> sel
    csv1 --> parse
    csv2 --> parse
    got --> combine
    parse --> combine

    classDef store fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a
    classDef muted fill:#f3f4f6,stroke:#9ca3af,color:#6b7280
    class csv1,csv2,prev,out store
    class skip,err muted

Visited URLs are remembered between runs, so an interrupted crawl resumes instead of starting over. Publication counts come from CSV exports rather than scraping.

Loading into Qdrant

Loading passes through four stages, and two of them buffer independently: up to 64 chunks are embedded per request, while points are written in batches of 128. Splitting them matters because the two calls have very different costs — embedding is a paid network round trip, an upsert is a local write — so neither should be paced by the other.

flowchart TB
    raw[("csrankings_TIMESTAMP.json")]

    subgraph docs["1 · Documents"]
        direction TB
        validate["DataModel.model_validate"]
        gate{"Has profile text<br/>and under 5 MB?"}
        drop["Skipped"]
        doc["AdvisorDocument<br/>cleaned text + metadata"]
        validate --> gate
        gate -- no --> drop
        gate -- yes --> doc
    end

    subgraph split["2 · Chunks"]
        direction TB
        tok["tiktoken cl100k_base"]
        win["512-token windows<br/>64-token overlap"]
        payload["Payload per chunk<br/>name · university · areas<br/>total_papers · homepage<br/>chunk_index · text"]
        tok --> win --> payload
    end

    subgraph embedding["3 · Embedding batch"]
        direction TB
        buf1["Buffer up to 64 chunks"]
        request["One embeddings request"]
        order["Re-sort by echoed index"]
        vec["1536-dim vectors"]
        buf1 --> request --> order --> vec
    end

    subgraph upsert["4 · Upsert batch"]
        direction TB
        point["PointStruct<br/>uuid4 + vector + payload"]
        buf2["Buffer up to 128 points"]
        ensure["Create collection if absent<br/>size 1536 · cosine"]
        push["client.upsert"]
        point --> buf2 --> ensure --> push
    end

    store[("Qdrant collection 'advisors'<br/>18,333 points")]

    raw --> validate
    doc --> tok
    payload --> buf1
    vec --> point
    push --> store

    classDef store fill:#dbeafe,stroke:#3b82f6,color:#1e3a8a
    classDef muted fill:#f3f4f6,stroke:#9ca3af,color:#6b7280
    class raw,store store
    class drop muted

Each chunk carries its advisor's metadata, so a search result identifies the person without a second lookup.

3,588

advisors indexed, 1 skipped for size

18,333

chunks, about 5 per advisor

112

universities represented

Deployment

The target topology on Azure. Qdrant needs a persistent volume, so it cannot scale to zero the way the API can. Supabase stays external — moving authentication to Entra External ID is deliberately out of scope for now.

flowchart TB
    subgraph azure["Azure subscription (target)"]
        direction TB
        swa["Static Web Apps<br/>Astro frontend"]
        aca["Container Apps<br/>FastAPI"]
        qdb["Container Apps + Azure Files<br/>Qdrant, persistent volume"]
        aoai["Azure OpenAI<br/>chat + embedding deployments"]
        kv["Key Vault<br/>secrets"]

        swa -- "/api proxy" --> aca
        aca --> qdb
        aca --> aoai
        aca -. reads .-> kv
    end

    supabase["Supabase<br/>auth, stays external"]
    swa -.-> supabase

    classDef ext fill:#fef3c7,stroke:#f59e0b,color:#78350f
    class supabase ext

Planned topology. Development currently runs the same stack locally with Docker and Qdrant on port 6333.

Models used

Recommendations and explanations: gpt-5.4-mini
Embeddings, 1536 dimensions: text-embedding-3-small

Every worked example on this site — the suggested prompts and the sample CV result — was produced by a real run against the full index with gpt-5.4-mini, then cached so the demo costs nothing to serve.

Advisor data is derived from public CSRankings faculty and publication records. Application requirements shown on recommendation cards are generated by a language model and must be verified against the department's own pages.