When is test automation worth it? Selenium vs Playwright vs Appium

“Should we set up test automation?” is one of the questions small teams and product owners ask most often. The answer is neither “absolutely” nor “don't bother”. Put in the right place, automation saves hours on every release; put in the wrong place, it turns into a pile of unmaintained tests that are always red and that nobody looks at anymore. In this article I explain in practical terms what manual and automated testing cover, when automation pays for itself, how to choose between Selenium, Playwright and Appium, and where to start.

What do manual and automated testing cover?

Manual testing means a person actually uses the application and checks it. The human eye catches odd design details, confusing copy and flows that “work but make no sense”. For exploratory testing, usability reviews and the first check of a new feature, manual testing is irreplaceable.

Automated testing means code runs predefined steps again and again, exactly the same way. It never gets tired, never skips a step and can run dozens of scenarios in minutes. But it only checks what it has been told to check: it does not “notice” new bugs, it verifies whether known behaviour still works.

In short, the two are not rivals but complements: automation takes over repetitive checks so people can spend their time where real thinking is needed.

When does automation pay off?

Every test you write is an investment: writing it takes time, and updating it as the application changes takes time too. That investment usually pays off when:

• You test the same flow often. If you click through login, sign-up, cart and checkout by hand on every release, those steps are the first candidates for automation.
• Some flows are expensive to break. A cart that cannot take payments or a broken password reset means lost money and lost customers.
• You release frequently. A team shipping several times a week cannot afford a long manual checklist every time.
• You support multiple browsers or devices. Trying the same scenario by hand in Chrome, Firefox and Safari gets tedious fast.
• The product has matured. In an early prototype whose interface changes every week, tests keep breaking; once the core flows settle, automation becomes far more efficient.

Conversely, for a one-off campaign page, a very early MVP or a visual design review, manual testing is usually the more sensible choice.

Choosing a tool: Selenium, Playwright, Appium

Selenium

Selenium is the most established browser automation tool and the foundation of the W3C WebDriver standard. It supports many languages, including Java, Python, C#, JavaScript and Ruby, and Selenium Grid lets you run tests in parallel across multiple machines. If your team already works in Java or C#, or you have an existing enterprise test setup, Selenium is still a strong choice. The downside is that you often have to build waiting logic and reporting yourself.

Playwright

Playwright is a newer tool developed by Microsoft. It drives the Chromium, Firefox and WebKit engines through a single API and supports TypeScript/JavaScript, Python, Java and .NET. Its biggest advantage is auto-waiting: it waits on its own until an element is actually ready to interact with, which significantly reduces flaky tests. A built-in test runner, parallel execution, screenshots, videos and a trace viewer for step-by-step inspection come out of the box. For a web project starting from scratch, Playwright is usually my recommendation.

Appium

If you have a native Android or iOS app rather than a website, Appium comes into play. Appium also uses the WebDriver protocol and runs tests on real devices and emulators through drivers such as UiAutomator2 for Android and XCUITest for iOS. Setup is more involved than with web tools and mobile tests are inherently slower, so it pays to be especially selective about the number of scenarios on mobile.

A small Playwright example

The test below verifies that a user can log in and see the dashboard. In real projects, the first tests I write are usually just as simple:

// tests/login.spec.ts
import { test, expect } from '@playwright/test';

test('user can log in and see the dashboard', async ({ page }) => {
  await page.goto('https://example.com/login');

  await page.getByLabel('Email').fill('demo@example.com');
  await page.getByLabel('Password').fill('correct-password');
  await page.getByRole('button', { name: 'Sign in' }).click();

  // Playwright waits automatically until these become true (or times out)
  await expect(page).toHaveURL(/dashboard/);
  await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
});

// Run: npx playwright test

Notice that instead of CSS classes it uses selectors based on what the user sees, such as getByRole and getByLabel. That way, tests break less often when the design changes.

CI integration

Tests show their real value when they run automatically, without anyone having to remember them. You add a step to a CI system such as GitHub Actions or GitLab CI that runs the tests on every pull request or every push to the main branch; if a test fails, the merge is blocked and the report, screenshots or trace files are kept as artifacts. Playwright offers a ready-made CI guide and sample configurations for this, and the initial setup takes less than an hour in most projects.

Common mistakes

• Trying to test everything. Hundreds of end-to-end tests make for a slow suite that is hard to maintain. It is healthier to cover critical flows end to end and details with unit and API tests.
• Ignoring flaky tests. A test that sometimes passes and sometimes fails destroys the team's trust in test results. Use condition-based waits instead of fixed sleep calls, and fix a flaky test right away, or disable it temporarily and log it.
• Fragile selectors. Long XPaths or auto-generated CSS classes break with the smallest design change. Prefer accessible roles, labels or dedicated data-testid attributes.
• Tests that depend on each other. Each test should prepare its own data and be able to run on its own; otherwise everything collapses when the order changes.
• Ignoring maintenance cost. Test code is code: it has to stay readable and be updated as the application changes.

How to start

1. List the three to five critical flows whose failure would cost you the most: usually login, sign-up, payment and the core feature.
2. Write these flows down step by step in plain text first; this also becomes your manual test checklist.
3. Automate those scenarios with Playwright for web or Appium for mobile.
4. Hook the tests into CI and run them on every change.
5. Grow the suite gradually as real bugs are caught or new critical features are added.

A small but reliable test suite is always worth more than a large one that nobody trusts.

Conclusion

Test automation is not a goal in itself; it is a tool that lets you ship with peace of mind. With the right flows and the right tool, it pays for itself quickly, even for small teams. If you would like to work out together which scenarios in your project are worth automating, or make your existing tests more reliable, feel free to get in touch.