Title: New Unit Tests Need to Fail (Running the Old Code)
Author: Alex Kirk
Published: October 9, 2023
Last modified: October 26, 2023

---

# New Unit Tests Need to Fail (Running the Old Code)

October 9, 2023

When possible, I very much recommend adding unit tests to a Pull Request when you
fix something as a way to prevent that it breaks again in future. This is no news.

One important step of adding that unit test, though, is to make sure it actually
tests the bug you are fixing. Specifically, this means that you need to test that
fact like this:**Keep the new unit test, undo the other code changes. The unit test
now needs to fail.If your newly introduced unit test still passes, then you didn’t
expose the bug in the test. A simple check but often neglected.

#### A tooling solution

Here is how you can automatically test this in Github. I have created a sample repo
with 3 pull requests [https://github.com/akirk/unit-test-failure/pulls](https://github.com/akirk/unit-test-failure/pulls):

![3 Pull requests of which one fails the unit tests](https://alex.kirk.at/wp-content/
uploads/sites/2/2023/10/unit-test-fail-3-prs-1.png)

You can see [the first one](https://github.com/akirk/unit-test-failure/pull/3) failed:

![A screenshot of a Github Action that has a unit test passing when it shouldn't](
https://alex.kirk.at/wp-content/uploads/sites/2/2023/10/old-code-passes-new-tests.
png)

The problem is that the old code already passes [the new tests](https://github.com/akirk/unit-test-failure/pull/3/files).
This means the tests don’t test the change.

The [second one passes](https://github.com/akirk/unit-test-failure/pull/2) but only
because [no new tests were added](https://github.com/akirk/unit-test-failure/pull/2/files)
at all.

![A screenshot of a Github Action that skipps testing without the unit test changes
because no new unit test was added](https://alex.kirk.at/wp-content/uploads/sites/
2/2023/10/no-unit-tests-added.png)

And the [third one does it correctly](https://github.com/akirk/unit-test-failure/pull/1).
The old code fails [the new tests](https://github.com/akirk/unit-test-failure/pull/1/files)
as desired:

![A screenshot of a Github Action that has correctly fails the unit test when running
the old code](https://alex.kirk.at/wp-content/uploads/sites/2/2023/10/unit-tests-
failed-as-expected.png)

Here is [the code for the Github Action to test this](https://github.com/akirk/unit-test-failure/blob/main/.github/workflows/unit-test-validation.yml):

    ```wp-block-code
    name: Pull Request Unit Test Validation

    on:
      pull_request:
        types:
          - opened
          - synchronize

    jobs:
      test-changes:
        runs-on: ubuntu-latest

        steps:
          - name: Checkout code
            uses: actions/checkout@v2

          - name: Set up Python
            uses: actions/setup-python@v2
            with:
              python-version: 3.9

          - name: Run unit tests with changes
            run: python -m unittest discover -s tests

      test-without-changes:
        needs: test-changes
        runs-on: ubuntu-latest
        if: ${{ github.event_name == 'pull_request' }}

        steps:
          - name: PR commits + 1
            run: echo "PR_FETCH_DEPTH=$(( ${{ github.event.pull_request.commits }} + 1 ))" >> "${GITHUB_ENV}"

          - name: Checkout PR branch and all PR commits
            uses: actions/checkout@v3
            with:
              ref: ${{ github.event.pull_request.head.sha }}
              fetch-depth: ${{ env.PR_FETCH_DEPTH }}

          - name: Set up Python
            uses: actions/setup-python@v2
            with:
              python-version: 3.9

          - name: 'Fetch the other branch with enough history for a common merge-base commit'
            run: git fetch origin ${{ github.event.pull_request.base.ref }}

          - name: Check if tests were changed
            run: |
              git restore --source=$(git merge-base origin/${{ github.event.pull_request.base.ref }} HEAD) --worktree tests/
              if git diff --quiet; then
                echo "TESTS_CHANGED=0" >> "${GITHUB_ENV}"
              else
                echo "TESTS_CHANGED=1" >> "${GITHUB_ENV}"
              fi
              git restore .

          - name: Revert code changes (excluding tests)
            run: git restore --source=$(git merge-base origin/${{ github.event.pull_request.base.ref }} HEAD) --worktree :^tests/
            if: ${{ env.TESTS_CHANGED }} == '1'

          - name: Run unit tests without changes
            run: |
              if [[ ${{ env.TESTS_CHANGED }} -eq 0 ]]; then
                echo "No unit test changes detected, skipping"
                exit 0
              fi

              if python -m unittest discover -s tests; then
                echo "Unit test should not pass"
                exit 1
              else
                echo "Unit test failed as expected"
              fi
    ```

Thanks @jesusamieiro-com for pointing out that you need to be careful with this 
for code that adds new test coverage. Probably best to use a Github label to activate
or deactivate this action.

[Code](https://alex.kirk.at/category/code/)

[Unit Tests](https://alex.kirk.at/tag/unit-tests/)

Read this next

[Resuming ChatGPT chats in CLI](https://alex.kirk.at/2023/09/25/resuming-chatgpt-chats-in-cli/)

### Leave a Reply 󠀁[Cancel reply](https://alex.kirk.at/2023/10/09/new-unit-tests-need-to-fail-running-the-old-code/?output_format=md#respond)󠁿

Only people in [my network](https://alex.kirk.at/friends/) can comment.

This site uses Akismet to reduce spam. [Learn how your comment data is processed.](https://akismet.com/privacy/)