Distributing a Python command line application
In this article I show how to create a minimal Python command line application, called ‘bootstrap’. I describe how to set it up for publication on PyPI, after which the user can conveniently install it via pip install bootstrap
. The installation immediately makes the ‘bootstrap’ command available to the user — for convenient invocation on Unix as well as on Windows. I show how to make the example application live within a proper package structure and how to make it callable and testable in different convenient ways. On Python 2 and Python 3.
Background
There are many ways to achieve the same thing. In the paragraphs below, I try to give proper advice, including current official recommendations, and schemes well-established in the Python community. One thing you need to know, and probably already realized yourself, is that Python packaging and package distribution can be quite tedious. In the past years, the recommendations for doing things “the right way” have often changed. Finally, however, it looks like we have something definite which I can base my article on.
Besides using the right tools, such as setuptools
(instead of distribute) and twine
, there is a lot of tension hidden in the details of the file/directory structure, and in the way you organize your application in terms of packages and modules. When do you need absolute or relative imports? What would be a convenient entry point for your application? For which parts do you need a wrapper? What is the right way to test the application without installing it? I do not deeply want to go into all these details in this article, but rather present a working solution. Just be sure that I have consulted official docs and guidelines, and taken a deeper look into how various established applications (such as sphinx, coverage, pep8,pylint) are set up in this regard. I have also consulted many answers on StackOverflow (e.g.this, this, this, this, and this), and finally implemented things myself (also here).
For this article, I try to break down all this valuable input to a minimal bare bones bootstrap project structure that should get you going. I try to reduce complexity, to avoid confusing constructs, and to not talk about difficulties anymore, from here on. The outcome is a very short and simple thing, really.
File structure
I recommend the following structure:
rootdir/ test/ docs/ bootstrap/ __init__.py bootstrap.py stuff.py main.py bootstrap-runner.py setup.py
Might look random in parts, but it is not. Clarification:
- All relevant application code is stored within the
bootstrap
package (which is thebootstrap/
directory containing the__init__.py
file). setup.py
contains instructions forsetuptools
. Below, I describe how to configure it in a way so thatsetuptools
creates an executable upon installation.bootstrap-runner.py
is just a simple wrapper script that allows for direct execution of the command line application from within therootdir/
(without installing).bootstrap/main.py
is the module containing a function which is the application’s entry point.bootstrap/stuff.py
is just an example for another module containing application logic, which can be imported frommain.py
bootstrap/bootstrap.py
is essentially not different fromstuff.py
. It should just demonstrate that it is fine to have a module within a package having the same name as the package.
File contents: bootstrap package
The contents of the files in the bootstrap
package, i.e. the application logic.
__init__.py
:
This file makes the bootstrap
directory a package. In simple cases, it can be left empty. We make use of that, leave it empty.
main.py
:
# -*- coding: utf-8 -*- """bootstrap: sample command line application""" from .stuff import Stuff from .bootstrap import Boo __version__ = "0.1.0" def main(): print("Welcome to '%s', version %s." % (__doc__, __version__)) print(Stuff) print(Boo()) if __name__ == "__main__": main()
As stated above, this module contains the function which is the main entry point to our application. We commonly call this function main()
. This main()
function is not called by importing the module, it is only called when
- the
bootstrap.main
module is executed as a script - or when or when
main()
is called directly from an external module.
This gives us great control, and further below I will make use of both cases.
Some more things worth discussing in the main.py
module:
- The module imports from other modules in the package. Therefore it uses relative imports. Implicit relative imports are forbidden in Python 3.
from .stuff import Stuff
is an explicit relative import, which you should make use of whenever possible. - People often define
__version__
in__init__.py
. Here, we define it inmain.py
, because it is simpler to access from withinmain.py
;-).
bootstrap.py
:
# -*- coding: utf-8 -*- """bootstrap.bootstrap: bootstrap module within the bootstrap package.""" from .stuff import Stuff class Boo(Stuff): pass
stuff.py
:
# -*- coding: utf-8 -*- """bootstrap.stuff: stuff module within the bootstrap package.""" class Stuff(object): pass
As you can see, the bootstrap.bootstrap
and bootstrap.stuff
modules define custom classes. Once again, bootstrap.bootstrap
contains an explicit relative import.
Executing the application: running the entry point function
You might be tempted to perform a $ python main.py
, which would fail with ValueError: Attempted relative import in non-package
. Is something wrong with the file structure or imports? No, is not. The invocation is wrong.
The right thing is to cd
into the project’s root directory, and then execute
$ python -m bootstrap.main
Output:
Welcome to 'bootstrap: sample command line application', version 0.1.0. <class 'bootstrap.stuff.Stuff'> <bootstrap.bootstrap.Boo object at 0x7fcb4d16b890>
python -m bootstrap.main
is doing what I wrote above:
main()
[...] is only called when thebootstrap.main
module is executed as a script
Does this look unusual to you? Well, this is not a 1-file-Python-script anymore. You are designing a package, and Python packages have special behavior. This is normal. However, there is a very straight-forward way to achieve the “normal” behavior that you are used to. That is what the convenience wrapper bootstrap-runner.py
is good for. Its content:
#!/usr/bin/env python # -*- coding: utf-8 -*- """ Convenience wrapper for testing development version of bootstrap. """ from bootstrap.main import main if __name__ == '__main__': main()
Should be self-explanatory, still: it imports the entry point function main
from modulebootstrap.main
and — if executed by itself as a script — invokes this function. Hence, you can use this bootstrap-runner.py
by itself as a normal script, as the command line frontend to your application. Set permissions via $ chmod u+x bootstrap-runner.py
and execute it:
$ ./bootstrap-runner.py Welcome to 'bootstrap: sample command line application', version 0.1.0. <class 'bootstrap.stuff.Stuff'> <bootstrap.bootstrap.Boo object at 0x7fa7110c48d0>
Straight-forward, right? You can now use bootstrap-runner.py
for testing purposes.
Preparing setup.py
Code upfront:
# -*- coding: utf-8 -*- import re from setuptools import setup version = re.search( '^__version__\s*=\s*"(.*)"', open('bootstrap/main.py').read(), re.M ).group(1) setup( name = "cmdline-bootstrap", packages = ["bootstrap"], entry_points = { "console_scripts": ['bootstrap = bootstrap.main:main'] }, version = version, description = "Python command line application bare bones template.", author = "Jan-Philip Gehrcke", author_email = "jgehrcke@googlemail.com", url = "http://gehrcke.de/2014/02/distributing-a-python-command-line-application", )
Some things to discuss:
- Might appear trivial, but
from setuptools import setup
is the current recommended way to go. - Your
setup.py
should not import your package for reading the version number. This fails for the end-user. Instead, always read it directly. In this case, I used regular expressions for extracting it. This is up to you. But never import your own module/package. - The
setup
function has many more useful arguments than shown here. For a serious project read the docs and make proper use oflong_description, author, classifiers, platform
, etc. - I have called the propject
cmdline-bootstrap
here instead of justbootstrap
, because I will really upload this to PyPI later on. And “bootstrap”, although still free, is just too much of a popular name to use it for something that small.
The essential arguments here are packages
and entry_points
. packages = ["bootstrap"]
tells setuptools that we want to install our bootstrap
package to the user’s site-packages
directory. The "console_scripts": ["bootstrap = bootstrap.main:main"]
line instructs setuptools to generate a script called bootstrap
. This script will invoke bootstrap.main:main
, i.e. the main
function of our bootstrap.main
module. This is the same idea as realized within bootstrap-runner.py
— the difference is that setuptools automatically creates this wrapper in the user’s file system when she/he installs bootstrap via pip install bootstrap
. setuptools places this wrapper into a directory that is in the user’s PATH, i.e. it immediately makes thebootstrap
command available to the user. Also on Windows (there, a small .exe file is created in something like C:\Python27\Scripts
) ;-).
Testing the setup
We use virtualenv
to reproduce what users see. Once, for Python 2(.7), once for Python 3(.3). Create both environments:
$ virtualenv --python=/path/to/python27 venvpy27 ... $ virtualenv --python=/path/to/python33 venvpy33 ...
Activate the 2.7 environment, and install the bootstrap application:
$ source venvpy27/bin/activate $ python setup.py install running install running bdist_egg running egg_info [...] Installed /xxx/venvpy27/lib/python2.7/site-packages/bootstrap-0.1.0-py2.7.egg Processing dependencies for bootstrap==0.1.0 Finished processing dependencies for bootstrap==0.1.0
See if (and where) the command has been created:
$ command -v bootstrap /xxx/venvpy27/bin/bootstrap
Try it:
$ bootstrap Welcome to 'bootstrap: sample command line application', version 0.1.0. <class 'bootstrap.stuff.Stuff'> <bootstrap.bootstrap.Boo object at 0x7f5577633890>
Great. Repeat the same steps for venvpy33
, and validate:
$ command -v bootstrap /xxx/venvpy33/bin/bootstrap $ bootstrap Welcome to 'bootstrap: sample command line application', version 0.1.0. <class 'bootstrap.stuff.Stuff'> <bootstrap.bootstrap.Boo object at 0x7f0a682dc390>
A note on automated tests
In the test/
directory you can set up automated tests for your application. You can always directly import the development version of your modules from e.g. test/test_api.py
, if you modify sys.path
:
sys.path.insert(0, os.path.abspath('..')) from bootstrap.stuff import Stuff
If you need to directly test the command line interface of your application, then bootstrap-runner.py
is your friend. You can easily invoke it from e.g. test/test_cmdline.py
via thesubprocess
module.
Upload your distribution file to PyPI
Create a source distribution of your project, by default this is a gzipped tarball:
$ python setup.py sdist $ /bin/ls dist cmdline-bootstrap-0.1.0.tar.gz
Register your project with PyPI. Then use twine to upload your project (twine is still to be improved!):
$ pip install twine $ twine upload dist/cmdline-bootstrap-0.1.0.tar.gz Uploading distributions to https://pypi.python.org/pypi Uploading cmdline-bootstrap-0.1.0.tar.gz Finished
Final test: install from PyPI
Create another virtual environment, activate it, install cmdline-bootstrap from PyPI and execute it:
$ virtualenv --python=/xxx/bin/python3.3 venvpy33test ... $ source venvpy33test/bin/activate $ bootstrap bash: bootstrap: command not found $ pip install cmdline-bootstrap Downloading/unpacking cmdline-bootstrap Downloading cmdline-bootstrap-0.1.0.tar.gz Running setup.py egg_info for package cmdline-bootstrap Installing collected packages: cmdline-bootstrap Running setup.py install for cmdline-bootstrap Installing bootstrap script to /xxx/venvpy33test/bin Successfully installed cmdline-bootstrap Cleaning up... $ bootstrap Welcome to 'bootstrap: sample command line application', version 0.1.0. <class 'bootstrap.stuff.Stuff'> <bootstrap.bootstrap.Boo object at 0x7fafd114e550>
'security_downloads' 카테고리의 다른 글
개인정보 법·제도 개선방안 연구* (0) | 2014.03.22 |
---|---|
What’s New In Python 3.4 (0) | 2014.03.21 |
What is Django and Steps to install Django (0) | 2014.03.21 |
PyFilesystem (0) | 2014.03.21 |
How To Dissect Android Flappy Bird Malware (0) | 2014.03.21 |