01 / BEFORE THE TOOL
Write down the expected answer.
For our first meeting-notes test, we wrote a fictional client conversation and an answer key before uploading anything. The answer key lists eight actions, three decisions, two unresolved questions, and claims the assistant should not make.
We deliberately include changes and refusals: a replaced deadline, an unapproved advertising budget, and a feature that is still undecided. This checks whether a summary preserves the final position, rather than simply listing everything mentioned.
Only the recording went into Fireflies. The tool did not receive our expected answers or scoring notes.
02 / DURING THE RUN
Keep the original output.
We record the input format, language, account state, selected summary template, execution date, and anything that interrupts the test. We preserve the original transcript and notes before editing, alongside screenshots of important results.
For publication, we select screenshot areas that exclude account information. We do not rewrite product output inside a screenshot. If we quote or paraphrase the output in a diagram or table, we label the interpretation.
Our first run used an English WAV file lasting 3 minutes 38 seconds, generated with one Microsoft Zira Desktop voice. The Fireflies interface used English (Global) and General Summary. Trial offers were declined.
A second 4:01 WAV used the same script with Zira for Emma and Microsoft Frank (nl-NL) reading English for Liam, with 350 ms pauses. We recorded the combined-notes scoring rules before this second upload and applied them to the unedited output. Voice, pronunciation and timing changed together; this was not an isolated speaker-count experiment. See the second-test evidence.
03 / AFTER THE RUN
A task needs more than a mention.
In our first assessment, we read Notes and Action items together. A fully correct task needs its action, owner, agreed deadline when one exists, and material restrictions. A conflicting owner elsewhere in the notes means it does not pass.
The action
Was the agreed work actually captured?
The owner
Is the right person responsible, consistently?
The due date
Is the final agreed date present and correct?
The boundary
Did a proposal stay unapproved or undecided?
The script and answer key were prepared in advance. We formalized this combined-notes scoring definition during assessment; it was not a preregistered benchmark protocol. We report missing tasks, ownership problems, omitted or incorrect deadlines, and invented commitments separately.
Our first Fireflies result was eight tasks mentioned, with five fully correct under this definition. Two tasks had owner problems and two had missing deadlines; those categories overlap.
The second run retained all eight expected tasks and all seven agreed dates, but four tasks met the full completeness rule: two tasks had conflicting owners and two omitted material restrictions. We treat the missing Liam-only invitee and the missing exclusion of internal ad discussion as incomplete, without claiming any message was actually sent.
CRM / A SEPARATE SCENARIO
Test the record, the answer and the action separately.
Our three-tool SaaS CRM pilot uses fictional notes with four agreed actions and an answer key prepared before execution. An AI assistant operated the interfaces in Marvin’s test accounts. Pipedrive and Close used paid-tier trials; HubSpot used Free. We disclose those differences, the exact prompts and extra attempts.
Read-only AI output, a manually saved task and a configured automation are different results. We kept them separate, switched off the test rules after execution and did not send fictional customer email or invitations. A later HubSpot inbox connection displayed an automatic connection-test email step. One run does not establish long-term accuracy, plan equivalence or duplicate prevention.
04 / WHAT THE RESULT MEANS
The claim should fit the test.
Two short synthetic recordings cannot establish overall accuracy. It does not measure speaker separation in a real meeting, background-noise tolerance, accent coverage, long-term reliability, or time saved.
We leave unmeasured results unrated. We do not convert a task count into a product-wide percentage, assign a star rating after one run, or compare tools when one has not produced equivalent output.
Fathom authentication did not complete in our Windows environment. We recorded that onboarding experience, but it gives us no basis for judging the quality of Fathom’s summaries.
ON THE WORKBENCH
What comes next.
EDITORIAL APPROACH
Keep observations separate from promises.
The Tech Stack Lab is an early editorial project run by Marvin Smit, focused on practical AI software. Our first review is based on the limited synthetic tests described here. Provider claims, documentation, and our own results should always be distinguishable.
The current articles use standard provider links without affiliate tracking. See the full editorial policy and report a correction. If we add affiliate links later, we will disclose them near the recommendation. Untested features will not acquire a score simply because a commission is available.
Pricing and plan limits change. Where we have not completed a current plan audit, we say so and direct readers to the provider rather than displaying an unverified price.