May 20, 2025

DIY User Research: How to use Maze, Prolific and other tools for fast real-world feedback

I wanted to learn what my target audience's perception of AI risk was through asking lots of people very quickly. This, it turns out, is also classed under "user research" - a field where UX designers have developed a robust toolkit.

I stumbled through figuring out how to do this efficiently while getting quick, actionable results. After cycling through 5+ tools and running ~10 surveys with 20 participants each, I've identified what you should actually look for if you're doing this yourself without a research team.

Caveat: this isn't comprehensive research, just me trying out a bunch of stuff I stumbled on. This is also not sponsored (but maybe you want to?) Here's what I learned.

Start Here: Core Principles

Before diving into the details, here are the key principles I wish I'd known from the beginning:

Start small and iterate - Test with 5 participants before scaling to catch issues early
Participant quality trumps quantity - The right 20 people provide more insight than 500 wrong ones
Platform interfaces matter enormously - They can make or break your efficiency
Analysis tools are separate decisions - Don't assume the platform's built-in analysis will be sufficient

Choosing the right participant panel

Quality matters above all else

Getting feedback from the wrong people will lead you astray. You need filters to proxy for the people you want to hear from and to block repeat respondents across similar surveys. The specificity of available filters depends on where they source their “panels”, whether they’ve cultivated their own (this is rare!) or are using a third-party user recruitment platform like Prolific or User Interviews.

In the absence of filters, you'll need to set up custom screening questions, though this sometimes requires you to pay for the user's time to complete them.

While most platforms tout "AI quality checks", if you're unfamiliar with the panel source it's worth doing a small test run (5-10 participants) to verify response quality before committing a larger portion of your budget. This small investment can save you from wasting your entire research budget on low-quality data.

Timing affects participant demographics

Most platforms promise results within hours, but when you post your survey significantly affects who responds. If you post at 2am and get responses within 2 hours, you're likely getting night owls, international participants, or people who rush through surveys for extra income - not typically representative of your target audience.

Consider scheduling your survey release to align with when your target demographic is most likely to be active online.

Designing effective questions for better insights

Start small and iterate

If you haven't had much experience with designing surveys, it's crucial to start with ≤5 participants for your first survey to validate your question phrasing and order. Since most platforms deliver results within hours, you have time to refine your approach based on initial responses. Better to make mistakes with 5 participants than 500.

Also, question whether you need statistical significance before scaling up. Often, open-ended qualitative questions to 10-20 people reveal deeper insights than hundreds of quantitative responses. For tight budgets, this approach delivers more bang for your buck.

Features to get more depth

Most participants will give a few words or a single sentence in open question responses. To extract richer insights, I've found two approaches particularly effective:

Record participants' verbal responses to questions, then transcribe them (more time-intensive but often more natural)
Have an AI ask follow-up questions (a feature Maze has!) to probe deeper based on initial responses

I've consistently found that higher-quality participants also correlate with more insightful responses - another reason not to skimp on panel quality.

Selecting a survey interface

I was surprised that most user research platforms don’t have their own survey interface! They link to third-party platforms like Qualtrics or SurveyMonkey, sometimes requiring you to figure out integration yourself.

For whichever survey design interface you choose, check for these surprisingly rare but valuable features:

Ability to create randomised split tests
Straightforward creation of custom screeners (the logic here can be unexpectedly complicated)
Support for having participants engage with content (videos, articles, images) before answering questions
Ability to reference responses to previous questions for conditional survey flow

The right interface can save you hours of frustration and enable more sophisticated research designs that would otherwise require programming knowledge.

What's missing: better guidance on question design

As a novice in survey design, I would have greatly benefited from integrated guidance on question phrasing to avoid leading questions or create comprehensive option lists. This is a gap across all platforms I tested.

Instead, I relied heavily on ChatGPT and Claude to help refine my survey questions - a workaround I recommend to others until platforms improve this aspect.

Cost considerations

Survey costs typically have two components: participant compensation and platform fees. And a hidden third cost: your own time.

Participant compensation

For a 20-minute survey, cost ranged from $5 to$ 20 per participant depending on targeting specificity. This variation makes a huge difference when scaling up your research.

Some platforms allow you to review and approve each response before payment, which is invaluable if you're on a tight budget or concerned about response quality. This feature alone can save you from wasting money on useless data.

If you already have access to your target audience, many platforms also support bringing your own participants, saving you platform fees.

Platforms

Many platforms frustratingly require sales calls before you can even try their product. This might be because they want to upsell their research team or reporting add-ons or get you signed onto a contract. (I’m not sure)

Platform pricing models vary widely:

Subscription-based with monthly or annual contracts ( $100-$ 3000+/month)
Pay-as-you-go up to certain usage levels
Combined models with basic access plus upgrade fees

If you’re just getting started, I recommend starting with pay-as-you-go options until you've established your ongoing research needs.

Time

When evaluating platforms, I drastically underestimated how much time I'd spend on platform-specific tasks rather than actual research design.

Some platforms force you through an exhausting laundry list of decisions:

Creating enticing survey titles to attract participants
Comparing your survey name against existing studies to avoid confusion
Writing promotional descriptions that sound like ad copy
Setting and justifying compensation rates
Crafting welcome messages and custom instructions
Manually configuring question types that should have intelligent defaults
Navigating through multiple non-intuitive interfaces to accomplish simple tasks

These seemingly trivial decisions add up dramatically. Platforms which minimise these micro-decisions with sensible defaults, allow you to focus on the questions that actually matter to your research.

Time efficiency isn't just convenience—it's the difference between being able to conduct multiple research iterations or being stuck with configuration. When evaluating platforms, I now recommend doing a test run with the stopwatch running from "I want to create a survey" to "My survey is live with participants" to measure the true time cost.

Making sense of the data

After testing multiple platforms, I've concluded that built-in analysis tools are universally disappointing, particularly for qualitative data. You're better off exporting your data and using a dedicated AI tool. Here's my ranking of AI analysis options:

#3 Claude 3.7 Sonnet

Despite its strength in writing, Claude struggles with CSV files (likely timing out before processing completion). Not recommended for survey analysis.

#2 NotebookLM

Excellent for handling multiple audio or video files with automatic transcription. I found it valuable for identifying broad themes across dozens of files.

However, for CSV analysis and generating surprising insights, it's less impressive than my top choice.

#1 OpenAI’s o3

The most effective tool for meaningful insights from survey data. While it can't handle large numbers of files, it excels when fed the themes from NotebookLM along with your CSV data.

One caution: o3 often hallucinates when asked to extract representative quotes, so always verify these against your raw data.

Making the right choice

After all my trial and error, here's the approach I recommend for:

If you’re just starting out: Start with just 5 participants and scale up to at most 20 until you’re sure you need statistical significance.
If efficiency is your priority: I found that Maze offered the best end-to-end experience with minimal friction. (I’m not even sponsored!) But beware that after the free trial, the price can get quite steep if you’re on a tight budget!
If depth of insights matters most: You can also consider running moderated surveys like 1-1 interviews or focus groups, which I didn’t cover here.
For all cases: Plan to export your data and analyse it with OpenAI's o3 rather than relying on platform analytics

The most important lesson? The differences between platforms aren't just about features but about how they affect your entire workflow when you’re doing everything yourself. Seemingly small interface differences can save or cost you hours of precious time.

Whatever platform you choose, start small, iterate quickly, and prioritise participant quality over quantity. You'll get more actionable insights faster and with less frustration.