Binary distributing a python package

Python Packages are created by using setup.py. The setup.py describes all aspects of a package like name, version, description, files etc.

A pure python package is where the package contains of only python files. This is relatively easy to package and distribute since the python files by itself have no dependency on the Operating System.

A package that contains an Operating System dependent files is usually distributed using wheels. A wheel is generated by running the following command

$ python setup.py bdist_wheel

The output of the above command is a wheel file(.whl extension) generated under dist directory. The generated wheel file name contains many details.

A sample wheel file name is minty-0.0.4-cp27-cp27mu-linux_x86_64.whl

For having a single wheel file to cater to all the variants of linux, a os tag of manylinux1 is used.

To generate a wheel file for the manylinux1 , we need to generate the wheel file using a special docker container available at quay.io/pypa/manylinux1_x86_64.

This docker container allows us to generate 2 wheels one for the narrow unicode(abi tag-cp27m) and wide unicode (abi tag-cp27mu)

$ docker run -it -v$PWD:/src quay.io/pypa/manylinux1_x86_64

once inside the container, compile the sources, build the shared library and generate the wheels using the python in the container. A source distribution (sdist) is also built for the operating systems where the wheels are not present

$ cd /src
$ cd minty/mstplib

$ make clean_build
rm -f libmyextn.so csrc.o
make libmyextn.so
make[1]: Entering directory `/src/minty/mstplib'
gcc -fPIC -c csrc.c
gcc -shared csrc.o -lpthread -o libmyextn.so
make[1]: Leaving directory `/src/minty/mstplib'

$ cd /src
$ /opt/_internal/cpython-2.7.15-ucs4/bin/python setup.py bdist_wheel
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/minty
...
...

$ /opt/_internal/cpython-2.7.15-ucs2/bin/python setup.py bdist_wheel
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/minty
...
...
$ /opt/_internal/cpython-2.7.15-ucs2/bin/python setup.py sdist
running sdist
running egg_info
writing requirements to minty.egg-info/requires.txt
writing minty.egg-info/PKG-INFO
writing top-level names to minty.egg-info/top_level.txt
writing dependency_links to minty.egg-info/dependency_links.txt
reading manifest file 'minty.egg-info/SOURCES.txt'
writing manifest file 'minty.egg-info/SOURCES.txt'
...
...

The python in ucs4 creates a wheel minty-0.0.4-cp27-cp27mu-linux_x86_64.whl The python in ucs2 creates a wheel minty-0.0.4-cp27-cp27m-linux_x86_64.whl

These wheels cannot be uploaded as is to pypi for distribution. They need to be checked and repaired by a tool auditwheel with respect to third-party external dependencies.

auditwheel is installed in a python3 environment using pip install auditwheel. There are 2 auditwheel commands that can be used, show and repair. The show command tells us if there are any external versioned symbols.

$ pip install auditwheel
Collecting auditwheel
  Using cached https://files.pythonhosted.org/packages/ac/c8/6b5e135684b8617eff1c2ffe6ac509837de904b852df682d1cd1c15235e8/auditwheel-1.9.0-py3-none-any.whl
...
...
Installing collected packages: auditwheel
Successfully installed auditwheel-1.9.0

$ auditwheel show minty-0.0.4-cp27-cp27mu-linux_x86_64.whl 
minty-0.0.4-cp27-cp27mu-linux_x86_64.whl is consistent with the
following platform tag: "manylinux1_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libpthread.so.0 with versions
{'GLIBC_2.2.5'}, libc.so.6 with versions {'GLIBC_2.2.5'}

The following external shared libraries are required by the wheel:
{
    "libc.so.6": "/lib/x86_64-linux-gnu/libc-2.23.so",
    "libpthread.so.0": "/lib/x86_64-linux-gnu/libpthread-2.23.so"
}

$ auditwheel repair minty-0.0.4-cp27-cp27mu-linux_x86_64.whl 
Repairing minty-0.0.4-cp27-cp27mu-linux_x86_64.whl
Previous filename tags: linux_x86_64
New filename tags: manylinux1_x86_64
Previous WHEEL info tags: cp27-cp27mu-linux_x86_64
New WHEEL info tags: cp27-cp27mu-manylinux1_x86_64

The distribution of python packages usually happens by uploading to pypi.org. For uploading a package to pypi.org, an account has to create by registering here.

After the registration, the uploading is done by using a utility called twine. We supply the list of the files to be uploaded on the command line. Typically the ‘*.whl’ files and the source distribution minty-*.tar.gz, generated by python setup.py sdist

twine uses a file .pypirc in the home directory that contains the user name and password that were used during the registration above.

$ cat ~/.pypirc 
[distutils]
index-servers =
  pypi

[pypi]
username: xxxxx
password: yyyyyyyyy

$ twine upload -r pypi dist/wheelhouse/*.whl dist/minty-*.tar.gz

twine can be installed in a python3 environment and used to upload packages which are in python2 environment also.