Dayjob: Tax – evaluating AI agents on 50 professional tax assignments
- ID
- 40878
- Status
- summarized
- Published
- 28 Oct 2026, 11:46 PM
- Fetched
- 30 Oct 2026, 2:36 AM
- Provider
- Hacker News
- Category
- dev-community
- Original URL
- https://surgehq.ai/blog/dayjob-tax
- Source URL
- https://hnrss.org/best
Summary
- Score
- 7.2
- Created
- 30 Oct 2026, 2:36 AM
- Tags
- Audience
- developersai_ml_learnersai_agent_userssaas_startup_founders
What happened
Surge AI released DAYJOB: Tax, a benchmark of 50 tax-work environments containing 1,789 files across eight tax practice areas, where agents must go from a client request to a finished return, workpaper, or advice. In one case, an agent comparing an October 2026 vs January 2027 condo sale correctly applied the $250,000 home-sale exclusion but missed that moving out in August 2024 creates an eligibility deadline around August 2027; Astra missed it in all five attempts, while Opus caught it in all five.
Why it matters
If you build or buy agents for accounting, tax, legal, or other professional workflows, test for unstated consequential issues, not just correct calculations: the benchmark's failure mode is missing a deadline the client did not ask about. The concrete decision is to add evals where the agent must surface eligibility windows and conflicting-record checks before producing a deliverable; a correct date comparison is not enough.
Discussion angle
What implicit requirement should your agent surface before it answers, and how would you test that with a messy 1,789-file environment rather than a clean prompt?