J is for JSON validation - Python A to Z

Python A-Z is a blog series about Python. Each day, I share insights, ideas and examples for different parts of Python development that match with the letter of the day. Blaugust is an annual blogging festival in August where the goal is to write a blog post every day of the month.

When you need to load external data into your software, you want to make sure it behaves properly, ie. it’s in correct form. One tool to validate input data is Pydantic. This is a quick tutorial to show how it works.

Installing Pydantic

Set up your environment and install with

pip install pydantic

Alternatively, you can run these with uv:

uv run --with pydantic code.py

Simplified Pokédex data

For this demo, I’m using a simplified data set adapted from Purukitto’s Pokemon.json. I’ve saved this to a file pokedex.json.

[
  {
    "id": 1,
    "name": "Bulbasaur",
    "type": ["Grass", "Poison"],
    "stats": {
      "HP": 45,
      "Attack": 49,
      "Defense": 49,
      "Sp. Attack": 65,
      "Sp. Defense": 65,
      "Speed": 45
    },
    "height": "0.7 m",
    "weight": "6.9 kg"
  },
  {
    "id": 2,
    "name": "Ivysaur",
    "type": ["Grass", "Poison"],
    "stats": {
      "HP": 60,
      "Attack": 62,
      "Defense": 63,
      "Sp. Attack": 80,
      "Sp. Defense": 80,
      "Speed": 60
    },
    "height": "1 m",
    "weight": "13 kg"
  }
]

Building a schema

Step 1: Basic types

Let’s start with a first step. We’ll define a class that inherits from Pydantic’s BaseModel. These classes will be the way we define the schema and what powers Pydantic’s validation engines.

For the first step, we’ll use basic definitions.

A Pokémon has a couple of fields: an id, a name, a list of types, a dictionary of stats, height and weight. A Pokedex is a list of Pokémon.

I will be using this as a base for all future snippets in this post but will only share what’s changed later on. You can find full code examples in GitHub.

import json

from pydantic import BaseModel
from typing import List, TypeAlias


class Pokemon(BaseModel):
    """An entry in Pokedex"""

    id: int
    name: str
    type: List[str]
    stats: dict
    height: str
    weight: str


Pokedex: TypeAlias = List[Pokemon]


def read_pokedex(filename: str) -> Pokedex:
    pokedex = []
    with open(filename, "r") as dex:
        data = json.load(dex)
        for entry in data:
            pokedex.append(Pokemon(**entry))
    return pokedex


if __name__ == "__main__":
    pokedex = read_pokedex("pokedex.json")
    print(pokedex)

Step 2: More refined types

Python itself offers enough for the basic types (like int, str , List and dict in the example above) but Pydantic offers a lot more if we want to be more specific.

For example, our id field is always a positive integer. We can use PositiveInt from Pydantic to add an extra layer of validation.

from pydantic import BaseModel, PositiveInt

class Pokemon(BaseModel):
    """An entry in Pokedex"""

    id: PositiveInt
    name: str
    type: List[str]
    stats: dict
    height: str
    weight: str

Everything else stays the same but we import PositiveInt and change id to it.

Now, lets we add a third element to our JSON to see how this works. Here, I’ve introduced a mistake in data: id is now -3 instead of 3.

{
    "id": -3,
    "name": "Venusaur",
    "type": ["Grass", "Poison"],
    "stats": {
      "HP": 80,
      "Attack": 82,
      "Defense": 83,
      "Sp. Attack": 100,
      "Sp. Defense": 100,
      "Speed": 80
    },
    "height": "2 m",
    "weight": "100 kg"
  }

If we run our code from step 1, all runs smoothly. If we run the code from step 2, we get a validation error:

pydantic_core._pydantic_core.ValidationError: 1 validation error for Pokemon
id
  Input should be greater than 0 [type=greater_than, input_value=-3, input_type=int]

Remember to switch -3 to 3 before continuing.

Step 3: Renaming fields

Our stats field looks like this:

    "stats": {
      "HP": 80,
      "Attack": 82,
      "Defense": 83,
      "Sp. Attack": 100,
      "Sp. Defense": 100,
      "Speed": 80
    }

We can use Pydantic’s alias feature to rename them to a more pythonic versions and add an extra layer of validation.

from pydantic import BaseModel, PositiveInt, Field
from typing import List, TypeAlias


class Stats(BaseModel):
    hp: PositiveInt = Field(alias="HP")
    attack: PositiveInt = Field(alias="Attack")
    sp_attack: PositiveInt = Field(alias="Sp. Attack")
    sp_defense: PositiveInt = Field(alias="Sp. Defense")
    speed: PositiveInt = Field(alias="Speed")


class Pokemon(BaseModel):
    """An entry in Pokedex"""

    id: PositiveInt
    name: str
    type: List[str]
    stats: Stats
    height: str
    weight: str

Here, we add an import for Field and create a new validation model Stats. For each field, we define the key in JSON as alias= argument and use the new Stats model in our Pokemon definition.

We now have added validation for all the stats and they are in a form that’s easier to use within Python code.

Step 4: Parse data into usable format

Our height and weight data is in a string format:

    "height": "2 m",
    "weight": "100 kg"

It would be easier to use them if they were stored numerically. Good news! We can run code before or after validation to modify the values.

Let’s start by writing functions to turn these data points into numbers:

def parse_height(height: str) -> float:
    height = height.removesuffix(" m")
    return float(height)


def parse_weight(weight: str) -> float:
    weight = weight.removesuffix(" kg")
    return float(weight)

We remove the unit suffixes and convert the rest to a float.

To run this, we need a few extra bits.

from pydantic import BaseModel, BeforeValidator, Field, PositiveFloat, PositiveInt
from typing import Annotated, List, TypeAlias

class Pokemon(BaseModel):
    """An entry in Pokedex"""

    id: PositiveInt
    name: str
    type: List[str]
    stats: Stats
    height: Annotated[PositiveFloat, BeforeValidator(parse_height)]
    weight: Annotated[PositiveFloat, BeforeValidator(parse_weight)]

We import three new things: pydantic.BeforeValidator , pydantic.PositiveFloat and typing.Annotated. (If we want to run code after value is validated, we could use pydantic.AfterValidator).

We then define height and weight as annotated floats and want to run their corresponding functions.

Now we can validate that the result of our parsing is a positive float for both of them and our code has access to properly typed values.

If our data used mixed units, this would also help us normalise the data. Here, all heights are in meters and weights are in kilograms so we didn’t have to do any conversions.

‘Tis but a scratch!

I’ve given you a glimpse into Pydantic’s validation tools and how to get started but there’s so much more you can do. They have pretty good documentation that will help you define and refine your validators even further.

Writing validators is not only good for catching up rogue, misformed data but it forces you to be explicit about understanding your data before you venture into building your program’s business logic.