Distributing a Python command line application

Distributing a Python command line application

In this article I show how to create a minimal Python command line application, called ‘bootstrap’. I describe how to set it up for publication on PyPI, after which the user can conveniently install it via pip install bootstrap. The installation immediately makes the ‘bootstrap’ command available to the user — for convenient invocation on Unix as well as on Windows. I show how to make the example application live within a proper package structure and how to make it callable and testable in different convenient ways. On Python 2 and Python 3.

Background

There are many ways to achieve the same thing. In the paragraphs below, I try to give proper advice, including current official recommendations, and schemes well-established in the Python community. One thing you need to know, and probably already realized yourself, is that Python packaging and package distribution can be quite tedious. In the past years, the recommendations for doing things “the right way” have often changed. Finally, however, it looks like we have something definite which I can base my article on.

Besides using the right tools, such as setuptools (instead of distribute) and twine, there is a lot of tension hidden in the details of the file/directory structure, and in the way you organize your application in terms of packages and modules. When do you need absolute or relative imports? What would be a convenient entry point for your application? For which parts do you need a wrapper? What is the right way to test the application without installing it? I do not deeply want to go into all these details in this article, but rather present a working solution. Just be sure that I have consulted official docs and guidelines, and taken a deeper look into how various established applications (such as sphinx, coverage, pep8,pylint) are set up in this regard. I have also consulted many answers on StackOverflow (e.g.this, this, this, this, and this), and finally implemented things myself (also here).

For this article, I try to break down all this valuable input to a minimal bare bones bootstrap project structure that should get you going. I try to reduce complexity, to avoid confusing constructs, and to not talk about difficulties anymore, from here on. The outcome is a very short and simple thing, really.

File structure

I recommend the following structure:

rootdir/
    test/
    docs/
    bootstrap/
        __init__.py
        bootstrap.py
        stuff.py
        main.py
    bootstrap-runner.py
    setup.py

Might look random in parts, but it is not. Clarification:

• All relevant application code is stored within the bootstrap package (which is thebootstrap/ directory containing the __init__.py file).
• setup.py contains instructions for setuptools. Below, I describe how to configure it in a way so that setuptools creates an executable upon installation.
• bootstrap-runner.py is just a simple wrapper script that allows for direct execution of the command line application from within the rootdir/ (without installing).
• bootstrap/main.py is the module containing a function which is the application’s entry point.
• bootstrap/stuff.py is just an example for another module containing application logic, which can be imported from main.py
• bootstrap/bootstrap.py is essentially not different from stuff.py. It should just demonstrate that it is fine to have a module within a package having the same name as the package.

File contents: bootstrap package

The contents of the files in the bootstrap package, i.e. the application logic.

__init__.py:
This file makes the bootstrap directory a package. In simple cases, it can be left empty. We make use of that, leave it empty.

main.py:

# -*- coding: utf-8 -*-
 
"""bootstrap: sample command line application"""
 
from .stuff import Stuff
from .bootstrap import Boo
 
__version__ = "0.1.0"
 
def main():
    print("Welcome to '%s', version %s." % (__doc__, __version__))
    print(Stuff)
    print(Boo())
 
if __name__ == "__main__":
    main()

As stated above, this module contains the function which is the main entry point to our application. We commonly call this function main(). This main() function is not called by importing the module, it is only called when

• the bootstrap.main module is executed as a script
• or when or when main() is called directly from an external module.

This gives us great control, and further below I will make use of both cases.

Some more things worth discussing in the main.py module:

• The module imports from other modules in the package. Therefore it uses relative imports. Implicit relative imports are forbidden in Python 3. from .stuff import Stuff is an explicit relative import, which you should make use of whenever possible.
• People often define __version__ in __init__.py. Here, we define it in main.py, because it is simpler to access from within main.py ;-).

bootstrap.py:

# -*- coding: utf-8 -*-
 
"""bootstrap.bootstrap: bootstrap module within the bootstrap package."""
 
from .stuff import Stuff
 
class Boo(Stuff):
    pass

stuff.py:

# -*- coding: utf-8 -*-
 
"""bootstrap.stuff: stuff module within the bootstrap package."""
 
class Stuff(object):
    pass

As you can see, the bootstrap.bootstrap and bootstrap.stuff modules define custom classes. Once again, bootstrap.bootstrap contains an explicit relative import.

Executing the application: running the entry point function

You might be tempted to perform a $ python main.py, which would fail with ValueError: Attempted relative import in non-package. Is something wrong with the file structure or imports? No, is not. The invocation is wrong.

The right thing is to cd into the project’s root directory, and then execute

$ python -m bootstrap.main

Output:

Welcome to 'bootstrap: sample command line application', version 0.1.0.
<class 'bootstrap.stuff.Stuff'>
<bootstrap.bootstrap.Boo object at 0x7fcb4d16b890>

python -m bootstrap.main is doing what I wrote above:

main() [...] is only called when the bootstrap.main module is executed as a script

Does this look unusual to you? Well, this is not a 1-file-Python-script anymore. You are designing a package, and Python packages have special behavior. This is normal. However, there is a very straight-forward way to achieve the “normal” behavior that you are used to. That is what the convenience wrapper bootstrap-runner.py is good for. Its content:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
 
"""
Convenience wrapper for testing development version of bootstrap.
"""
 
from bootstrap.main import main
 
if __name__ == '__main__':
    main()

Should be self-explanatory, still: it imports the entry point function main from modulebootstrap.main and — if executed by itself as a script — invokes this function. Hence, you can use this bootstrap-runner.py by itself as a normal script, as the command line frontend to your application. Set permissions via $ chmod u+x bootstrap-runner.py and execute it:

$ ./bootstrap-runner.py 
Welcome to 'bootstrap: sample command line application', version 0.1.0.
<class 'bootstrap.stuff.Stuff'>
<bootstrap.bootstrap.Boo object at 0x7fa7110c48d0>

Straight-forward, right? You can now use bootstrap-runner.py for testing purposes.

Preparing setup.py

Code upfront:

# -*- coding: utf-8 -*-
 
import re
from setuptools import setup
 
version = re.search(
    '^__version__\s*=\s*"(.*)"',
    open('bootstrap/main.py').read(),
    re.M
    ).group(1)
 
setup(
    name = "cmdline-bootstrap",
    packages = ["bootstrap"],
    entry_points = {
        "console_scripts": ['bootstrap = bootstrap.main:main']
        },
    version = version,
    description = "Python command line application bare bones template.",
    author = "Jan-Philip Gehrcke",
    author_email = "jgehrcke@googlemail.com",
    url = "http://gehrcke.de/2014/02/distributing-a-python-command-line-application",
 
    )

Some things to discuss:

• Might appear trivial, but from setuptools import setup is the current recommended way to go.
• Your setup.py should not import your package for reading the version number. This fails for the end-user. Instead, always read it directly. In this case, I used regular expressions for extracting it. This is up to you. But never import your own module/package.
• The setup function has many more useful arguments than shown here. For a serious project read the docs and make proper use of long_description, author, classifiers, platform, etc.
• I have called the propject cmdline-bootstrap here instead of just bootstrap, because I will really upload this to PyPI later on. And “bootstrap”, although still free, is just too much of a popular name to use it for something that small.

The essential arguments here are packages and entry_points. packages = ["bootstrap"] tells setuptools that we want to install our bootstrap package to the user’s site-packages directory. The "console_scripts": ["bootstrap = bootstrap.main:main"] line instructs setuptools to generate a script called bootstrap. This script will invoke bootstrap.main:main, i.e. the mainfunction of our bootstrap.main module. This is the same idea as realized within bootstrap-runner.py — the difference is that setuptools automatically creates this wrapper in the user’s file system when she/he installs bootstrap via pip install bootstrap. setuptools places this wrapper into a directory that is in the user’s PATH, i.e. it immediately makes thebootstrap command available to the user. Also on Windows (there, a small .exe file is created in something like C:\Python27\Scripts) ;-).

Testing the setup

We use virtualenv to reproduce what users see. Once, for Python 2(.7), once for Python 3(.3). Create both environments:

$ virtualenv --python=/path/to/python27 venvpy27
...
$ virtualenv --python=/path/to/python33 venvpy33
...

Activate the 2.7 environment, and install the bootstrap application:

$ source venvpy27/bin/activate
$ python setup.py install
running install
running bdist_egg
running egg_info
[...]
Installed /xxx/venvpy27/lib/python2.7/site-packages/bootstrap-0.1.0-py2.7.egg
Processing dependencies for bootstrap==0.1.0
Finished processing dependencies for bootstrap==0.1.0

See if (and where) the command has been created:

$ command -v bootstrap
/xxx/venvpy27/bin/bootstrap

Try it:

$ bootstrap
Welcome to 'bootstrap: sample command line application', version 0.1.0.
<class 'bootstrap.stuff.Stuff'>
<bootstrap.bootstrap.Boo object at 0x7f5577633890>

Great. Repeat the same steps for venvpy33, and validate:

$ command -v bootstrap
/xxx/venvpy33/bin/bootstrap
$ bootstrap
Welcome to 'bootstrap: sample command line application', version 0.1.0.
<class 'bootstrap.stuff.Stuff'>
<bootstrap.bootstrap.Boo object at 0x7f0a682dc390>

A note on automated tests

In the test/ directory you can set up automated tests for your application. You can always directly import the development version of your modules from e.g. test/test_api.py, if you modify sys.path:

sys.path.insert(0, os.path.abspath('..'))
from bootstrap.stuff import Stuff

If you need to directly test the command line interface of your application, then bootstrap-runner.py is your friend. You can easily invoke it from e.g. test/test_cmdline.py via thesubprocess module.

Upload your distribution file to PyPI

Create a source distribution of your project, by default this is a gzipped tarball:

$ python setup.py sdist
$ /bin/ls dist
cmdline-bootstrap-0.1.0.tar.gz

Register your project with PyPI. Then use twine to upload your project (twine is still to be improved!):

$ pip install twine
$ twine upload dist/cmdline-bootstrap-0.1.0.tar.gz
Uploading distributions to https://pypi.python.org/pypi
Uploading cmdline-bootstrap-0.1.0.tar.gz
Finished

Final test: install from PyPI

Create another virtual environment, activate it, install cmdline-bootstrap from PyPI and execute it:

$ virtualenv --python=/xxx/bin/python3.3 venvpy33test
...
$ source venvpy33test/bin/activate
$ bootstrap
bash: bootstrap: command not found
$ pip install cmdline-bootstrap
Downloading/unpacking cmdline-bootstrap
  Downloading cmdline-bootstrap-0.1.0.tar.gz
  Running setup.py egg_info for package cmdline-bootstrap
 
Installing collected packages: cmdline-bootstrap
  Running setup.py install for cmdline-bootstrap
 
    Installing bootstrap script to /xxx/venvpy33test/bin
Successfully installed cmdline-bootstrap
Cleaning up...
$ bootstrap
Welcome to 'bootstrap: sample command line application', version 0.1.0.
<class 'bootstrap.stuff.Stuff'>
<bootstrap.bootstrap.Boo object at 0x7fafd114e550>