Custom card sort study tool

An API information-architecture study, and the AI-built tool that deleted its own data

May to June 2026

  • Research study
Role
Designer and study infrastructure owner, with the analysis and readout
What it is
A card sort tool I built with an AI app builder, and the study it ran with 28 developers
Result
Developers' groupings matched the primitive structure, not the product one: 5 of 9 clusters against 2.
Three overlapping windows of the card sort tool. One is a study with Cards, Categories, Questions, and Settings tabs and a list of API cards to add. One has an A/B test switch for two category sets. The front one is the results view on the Dendrogram tab, with nine colored cluster labels above a tree. The participant link is hidden.
The tool I built with an AI app builder: open, closed, and hybrid sorts, A/B tests between and within subjects, a preview mode, participant analytics, principal component analysis, and a results dashboard. The front window shows nine clusters at 50% agreement. The participant link is hidden. View full size: Three overlapping windows of the card sort tool

HubSpot has many public APIs. Should they be grouped by product, or by the primitives underneath: objects, events, communication, content? I built the tool above in about four weeks, and asked 28 developers.

What came out

The results overview from the card sort tool. Four figures run across the top: started, completed, a completion rate of 50%, and median time. A bar chart of responses per day, from May 20 to June 3, sits below them.
56 developers started and 28 finished, sorting 49 cards. Median time was just under 14 minutes. View full size: The results overview from the card sort tool
  1. 9 cardsCommerce Subscriptions, Invoices, Commerce Payments, Tickets, Line Items, Products, Carts, Object Schemas, Pipelines
  2. 4 cardsAssociations, Contacts, Quotes, Orders
  3. 6 cardsSMS / WhatsApp Messages, Communication Channels, Subscription Preferences, Campaigns, Marketing Email, Transactional Email
  4. 4 cardsEmails (Activity Log), Postal Mail, Calls, Conversation Inboxes
  5. 5 cardsMeetings, Notes, Tasks, Custom Events, Custom Timeline Events
  6. 9 cardsBlog Posts, Files, HubDB Tables, URL Redirects, Search, Landing Pages, Site Search, Forms, Lists
  7. 5 cardsProperties, Data Imports, Domains, Account Information, Users
  8. 2 cardsWorkflows, Custom Workflow Actions
  9. 6 cardsCalling Extensions, Video Conferencing Extensions, Webhooks, App Installs, Visitor Identification, CRM Cards
Nine clusters at 50% agreement from 28 developers. Five line up with the primitive structure and two with the product one. 9 of 28 developers made a Commerce group on their own.

The primitive-aligned structure won.

How it was built

Three side-by-side panels, one for each of the participant sorts that most other participants agreed with. Each panel lists API cards under group names such as Objects, Content, Events, and Communication.
My Python analysis scored each hypothesis against the sorts. The best match scored 0.565, moderate agreement. Shown: the three sorts most others agreed with, from before the data recovery. Objects, events, communication, and content show up in all three. View full size: Three side-by-side panels, one for each of the participant sorts that most other participants agreed with

The first build had five serious incidents in four weeks. A routine change to save questions deleted every question response. A week later the same cause took the card placements of 15 of 28 participants.

CriticalHighMediumRecoveryMay 14May 28Jun 1Jun 4Jun 10 to 12data recoveredanalysis re-runCriticalHighMediumMay 14May 28Jun 1Jun 4Jun 10 to 12Recoverydata recoveredanalysis re-run
The first build had five incidents in four weeks. Each sits at its date and severity, with recovery last.

I restored from a backup, remapped the rest by label with additive inserts, and re-ran the analysis. I also caught a count mismatch (12 participants against 28 responses) before the team accepted the results.

My rules for agents come from this: safe defaults in the schema, additive writes, a separate verifier, and no destructive action without a person saying yes. Later I rebuilt the platform with Claude Code.

AGENTS.md

Semi-autonomous. Reversible two-way-door actions (local edits, memory
consolidation, branch commits) run unattended. One-way doors (external
posts, pushes to shared repos, deletes, hook or permission changes) are
gated: park them in ## Pending review, never do them unattended.
The rule every recurring agent job of mine follows today. Deletes wait for me. Shortened. From my home setup.