Like it!

Join us on Facebook!

Like it!

Modules and packages: how to create a Python project

A quick and dirty tutorial on how to get things done.

Do you sweat every time you have to organize a bunch of new Python code? Should you make a module, a package, or both? Don't worry, I've been there thousand times. In this article I will show you the rationale behind a Python module, the differences with a package and how the two things interact together.

Python modules vs. Python packages

A module is a Python file which contains code, saved with the .py extension. Every time you write a function, a class or a statement and save it to a .py file you are actually creating a new module. A module can be executed by the Python interpreter directly, e.g. python script.py and thus called main module, or imported by other modules. Modules are a way to lay out your program in different files for easier maintenance.

When you have many modules in your project it's good practice to organize them into folders. For example, say I'm working on a very primitive game in Python called Fancy Game: I would like to structure the directory as follows:

fancy_game/
    models/
        player.py
        monster.py
    audio/
        mixer.py
        effects.py
        player.py
    graphics/
        renderer.py
        screen.py
    common/
        constants.py
    main.py

A package is simply a collection of Python modules organized into folders. In my Fancy Game the packages could be: models, audio, graphics and common. The fancy_game folder is not a package itself, because it is intended to be run directly by Python (i.e. python main.py). Sometimes you want to create a library instead to be imported in other Python programs, so the entire root folder would become a package too (made of many sub-packages).

Having a project or a library organized into packages is a good thing: a) your source code is even more modularized and b) packages provide protection against name clashes with other modules. We'll see why in a minute.

Turn a folder into a Python package

Python has to be instructed about which directory should become a package. To do this, simply add an empty file called __init__.py inside each desired folder. This is a special file used to mark directories on disk as Python package directories. So, my Fancy Game folder structure would be:

fancy_game/
    models/
        __init__.py        <--- new __init__.py file added
        player.py
        monster.py
    audio/
        __init__.py        <--- new __init__.py file added
        mixer.py
        effects.py
        player.py
    graphics/
        __init__.py        <--- new __init__.py file added
        renderer.py
        screen.py
    common/
        __init__.py        <--- new __init__.py file added
        constants.py
    main.py

Notice how there is no __init__.py in the root folder: this is because my game (i.e. main.py) is intended to be run directly from the Python interpreter. In case of a library, simply add the special file into the root directory as well.

Importing modules from packages

Now that the whole structure has been set up, the code inside main.py needs to import some modules from the various packages in order to make the game work. To import a module from a package you have to follow the dotted module name syntax. For example, in the main module I want to import the player module from the audio package:

# main.py
import audio.player

More generally, the rule is import [package].[module]. This also works in case you have nested packages: import [package1].[package2].[module] and so on.

Once imported, the module must be referenced with its full name. So if I want to use the function play_sound() from within the audio.player module I have to call it as audio.player.play_sound(). As mentioned above, this is a good way to avoid name clashes across different modules: I can easily import the module model.player without messing up with its homonym audio.player:

# main.py
import audio.player
import model.player

# Two modules with the same name: no problem
audio.player.play_sound()
model.player.run()

Importing modules from above

Sometimes a deep-buried module needs to import stuff from the upper level. For example, the audio.player module might need something inside common.constants. There are two ways of doing it:

  1. absolute import — import the module as if the importer is located in the root directory. Python is able to figure out the right path. For example, in audio.player just do import common.constants. This is my favorite option;
  2. relative import — use the formula from [module] import [name] with dots to indicate the current and parent packages involved in the import. For example, in audio.player you can call from .. import common.constants. One dot means the current package, two dots is up one level, three dots is up two levels and so on. I'm not a huge fan of this one, as relative imports break easily when you move modules around.

Shorten module names

Using long names such as models.monster.Skeleton is quite inconvenient. You can shorten a module name while importing it with an alias, for example:

# main.py
import models.monster as monster

Now models.monster is available as monster. Just keep in mind that this way might lead to name clashes across modules.

Sources

Alex Dzyoba - Hitchhiker's guide to the Python imports
Stack Overflow - What is __init__.py for?
Python docs - 6. Modules
Python reference - 7.11. The import statement

comments
flying sheep on January 27, 2019 at 14:11
> Python has to be instructed about which directory should become a package. To do this, simply add an empty file called __init__.py inside each desired folder.

This is simply wrong as of Python 3.0: Every directory with a alphanumeric name is a python package. No need for __init__.py files unless you want to actually put code in them.
Landon on January 27, 2019 at 15:11
Thank you, this is a great explanation.
Triangles on January 28, 2019 at 10:51
@flying sheep good catch, too bad it is not mentioned in the official documentation! This is the PEP:
https://www.python.org/dev/peps/pep-0420/
and this is a short explanation: https://stackoverflow.com/a/37140173/3296421
Thank you :)
Florian on January 29, 2019 at 22:45
Cool Post! Have you checked out PyScaffold to create a Python package structure easily? https://pyscaffold.org/
Fabien on January 30, 2019 at 22:09
That's a nice quick and dirty tutorial!
I would suggest a few precisions:

1) good practice in python 3 is to use *relative import* syntax for packages not installed as site packages and directly available to the execution script:

from . import audio.player # notice the "." that tells audio is in the root dir
from .audio import player # equivalent, but shorter name

See pep 328 for details (https://www.python.org/dev/peps/pep-0328/)

2) use import *from* to shorten the module name, and *as* to use a meaningful alias:

from .models import monster # no need for alias there
import .audio.effects as afx # short and usefull alias

See pep 221 for details on *import as* (https://www.python.org/dev/peps/pep-0221/)

3) If *main.py* is meant as an executable script for the game, the whole project should be a package, and main.py should not use relative import.
The reason why is that when you finally decide to distribute the fancy game, and install *main.py* as an executable scripts from setup.py, it will no longer be able to perform relative imports from this package.

For this use case, create a __init__.py for the *fancy_game* dir, and use:

from fancy_game import audio.player
from fancy_game.audio import effects as afx
from fancy_game.models import monster

Which will not break upon installation.

This is likely a bit beyond this tutorial, but mentioning this could prevent headaches :)
Tesfit on July 12, 2019 at 09:29
You explain it explicitly, thank you
Serob on September 02, 2019 at 21:28
What about sys.path-s? Because without adding project path to PYTHONPATH this approach doesn't work (at least on windows).
And see the question
https://stackoverflow.com/questions/57760296/can-not-import-package-created-in-pycharm
Kristy on September 16, 2020 at 08:53
Exactly, Serob! I am on Windows 10, and I don't want to copy a "util.py" file I made, to every single project that uses it. Why not be able to have that "util.py" off in some forgotten directory (nowhere near the current project's directory) and just import it? This tutorial assumes that all modules / packages used in an app are either:
(a) built-in modules / packages
(b) modules / packages located in the root directory of the app.
ABHIRAM ANIL on January 05, 2021 at 17:00
which refers to a collection of module that together cater to specific type of needs or application?
Antoine on April 20, 2021 at 19:30
Thank you! Very clearly explained. Super duper helpful :)
Daniel on January 06, 2022 at 23:50
Amazing! This is exactly the simple and effective tutorial I need to make my source code more modular and readable.

Thank you!