From d51059ac1cc2fcc54a572eecadfae7e2bb07ad70 Mon Sep 17 00:00:00 2001 From: alvyjudy Date: Tue, 26 May 2020 00:43:54 -0400 Subject: docs: WIP detailed guide on pkg discovery --- docs/userguide/package_discovery.txt | 101 ++++++++++++++++++++++------------- 1 file changed, 63 insertions(+), 38 deletions(-) diff --git a/docs/userguide/package_discovery.txt b/docs/userguide/package_discovery.txt index 8ba12cdf..45d88b60 100644 --- a/docs/userguide/package_discovery.txt +++ b/docs/userguide/package_discovery.txt @@ -1,65 +1,90 @@ +.. _`package_discovery`: + =================== Package Discovery =================== ``Setuptools`` provide powerful tools to handle package discovery, including -support for namespace package. The following explain how you include package -in your ``setup`` script:: +support for namespace package. Normally, you would specify the package to be +included manually in the following manner: + +.. code-block:: ini + + [options] + packages = + mypkg1 + mypkg2 + +.. code-block:: python setup( packages = ['mypkg1', 'mypkg2'] ) -To speed things up, we introduce two functions provided by setuptools:: +This can get tiresome reallly quickly. To speed things up, we introduce two +functions provided by setuptools: - from setuptools import find_packages +.. code-block:: ini -or:: + [options] + packages = find: + #or + packages = find_namespace: +.. code-block:: python + + from setuptools import find_packages + #or from setuptools import find_namespace_packages -Using ``find_packages()`` -------------------------- +Using ``find:`` (``find_packages``) +=================================== +Let's start with the first tool. ``find:`` (``find_packages``) takes a source +directory and two lists of package name patterns to exclude and include, and +then return a list of ``str`` representing the packages it could find. To use +it, consider the following directory + + mypkg/ + src/ + pkg1/__init__.py + pkg2/__init__.py + tests/__init__.py + setup.cfg #or setup.py -Let's start with the first tool. +To have your setup.cfg or setup.py to automatically include packages found +in ``src`` that starts with the name ``pkg`` and not ``tests``: -``find_packages()`` takes a source directory and two lists of package name -patterns to exclude and include. If omitted, the source directory defaults to -the same -directory as the setup script. Some projects use a ``src`` or ``lib`` -directory as the root of their source tree, and those projects would of course -use ``"src"`` or ``"lib"`` as the first argument to ``find_packages()``. (And -such projects also need something like ``package_dir={"": "src"}`` in their -``setup()`` arguments, but that's just a normal distutils thing.) +.. code-block:: ini -Anyway, ``find_packages()`` walks the target directory, filtering by inclusion -patterns, and finds Python packages (any directory). Packages are only -recognized if they include an ``__init__.py`` file. Finally, exclusion -patterns are applied to remove matching packages. + [options] + packages = find: + package_dir = + =src -Inclusion and exclusion patterns are package names, optionally including -wildcards. For -example, ``find_packages(exclude=["*.tests"])`` will exclude all packages whose -last name part is ``tests``. Or, ``find_packages(exclude=["*.tests", -"*.tests.*"])`` will also exclude any subpackages of packages named ``tests``, -but it still won't exclude a top-level ``tests`` package or the children -thereof. In fact, if you really want no ``tests`` packages at all, you'll need -something like this:: + [options.packages.find] + where = src + include = pkg* + exclude = tests - find_packages(exclude=["*.tests", "*.tests.*", "tests.*", "tests"]) +.. code-block:: python -in order to cover all the bases. Really, the exclusion patterns are intended -to cover simpler use cases than this, like excluding a single, specified -package and its subpackages. + setup( + #... + packages = find_packages( + where = 'src', + include = ['pkg*',], + exclude = ['tests',] + ), + package_dir = {"":"src"} + #... + ) -Regardless of the parameters, the ``find_packages()`` -function returns a list of package names suitable for use as the ``packages`` -argument to ``setup()``, and so is usually the easiest way to set that -argument in your setup script. Especially since it frees you from having to -remember to modify your setup script whenever your project grows additional -top-level packages or subpackages. +Of course the keywords presented here appear arbitary and the example given +doesn't apply to every other scenarios. For best understanding, we recommend +going to :ref:`keywords_ref`. +#####WIP below######### ``find_namespace_packages()`` ----------------------------- In Python 3.3+, ``setuptools`` also provides the ``find_namespace_packages`` variant -- cgit v1.2.3 From 9b87e896de30cda6003bb2f18c41c5f6c4db9d12 Mon Sep 17 00:00:00 2001 From: alvyjudy Date: Tue, 26 May 2020 13:27:34 -0400 Subject: docs: detail pkg discover guide and namespace pkg userguide/pkg_discovery.txt now covers find_package, find_namespace package and legacy use of namespace package creation in both setup.py and setup.cfg style --- docs/userguide/package_discovery.txt | 210 +++++++++++++++++------------------ 1 file changed, 105 insertions(+), 105 deletions(-) diff --git a/docs/userguide/package_discovery.txt b/docs/userguide/package_discovery.txt index 45d88b60..77d61770 100644 --- a/docs/userguide/package_discovery.txt +++ b/docs/userguide/package_discovery.txt @@ -1,8 +1,19 @@ .. _`package_discovery`: -=================== -Package Discovery -=================== +======================================== +Package Discovery and Namespace Package +======================================== + +.. note:: + a full specification for the keyword supplied to ``setup.cfg`` or + ``setup.py`` can be found at :ref:`keywords reference ` + +.. note:: + the examples provided here are only to demonstrate the functionality + introduced. More metadata and options arguments need to be supplied + if you want to replicate them on your system. If you are completely + new to setuptools, the :ref:`quickstart section ` is a good + place to start. ``Setuptools`` provide powerful tools to handle package discovery, including support for namespace package. Normally, you would specify the package to be @@ -38,22 +49,25 @@ functions provided by setuptools: from setuptools import find_namespace_packages -Using ``find:`` (``find_packages``) -=================================== +Using ``find:`` or ``find_packages`` +==================================== Let's start with the first tool. ``find:`` (``find_packages``) takes a source directory and two lists of package name patterns to exclude and include, and then return a list of ``str`` representing the packages it could find. To use it, consider the following directory +.. code-block:: bash + mypkg/ src/ pkg1/__init__.py pkg2/__init__.py - tests/__init__.py + additional/__init__.py + setup.cfg #or setup.py To have your setup.cfg or setup.py to automatically include packages found -in ``src`` that starts with the name ``pkg`` and not ``tests``: +in ``src`` that starts with the name ``pkg`` and not ``additional``: .. code-block:: ini @@ -65,7 +79,7 @@ in ``src`` that starts with the name ``pkg`` and not ``tests``: [options.packages.find] where = src include = pkg* - exclude = tests + exclude = additional .. code-block:: python @@ -80,127 +94,113 @@ in ``src`` that starts with the name ``pkg`` and not ``tests``: #... ) -Of course the keywords presented here appear arbitary and the example given -doesn't apply to every other scenarios. For best understanding, we recommend -going to :ref:`keywords_ref`. -#####WIP below######### -``find_namespace_packages()`` ------------------------------ -In Python 3.3+, ``setuptools`` also provides the ``find_namespace_packages`` variant -of ``find_packages``, which has the same function signature as -``find_packages``, but works with `PEP 420`_ compliant implicit namespace -packages. Here is a minimal setup script using ``find_namespace_packages``:: +Using ``find_namespace:`` or ``find_namespace_packages`` +======================================================== +``setuptools`` provides the ``find_namespace:`` (``find_namespace_packages``) +which behaves similarly to ``find:`` but works with namespace package. Before +diving in, it is important to have a good understanding of what namespace +packages are. Here is a quick recap: - from setuptools import setup, find_namespace_packages - setup( - name="HelloWorld", - version="0.1", - packages=find_namespace_packages(), - ) +Suppose you have two packages named as follows: +.. code-block:: bash -Keep in mind that according to PEP 420, you may have to either re-organize your -codebase a bit or define a few exclusions, as the definition of an implicit -namespace package is quite lenient, so for a project organized like so:: + /Users/Desktop/timmins/foo/__init__.py + /Library/timmins/bar/__init__.py +If both ``Desktop`` and ``Library`` are on your ``PYTHONPATH``, then a +namespace package called ``timmins`` will be created automatically for you when +you invoke the import mechanism, allowing you to accomplish the following - ├── namespace - │   └── mypackage - │   ├── __init__.py - │   └── mod1.py - ├── setup.py - └── tests - └── test_mod1.py +.. code-block:: python -A naive ``find_namespace_packages()`` would install both ``namespace.mypackage`` and a -top-level package called ``tests``! One way to avoid this problem is to use the -``include`` keyword to whitelist the packages to include, like so:: + >>> import timmins.foo + >>> import timmins.bar - from setuptools import setup, find_namespace_packages +as if there is only one ``timmins`` on your system. The two packages can then +be distributed separately and installed individually without affecting the +other one. Suppose you are packaging the ``foo`` part: - setup( - name="namespace.mypackage", - version="0.1", - packages=find_namespace_packages(include=["namespace.*"]) - ) +.. code-block:: bash + + foo/ + src/ + timmins/foo/__init__.py + setup.cfg # or setup.py + +and you want the ``foo`` to be automatically included, ``find:`` won't work +because timmins doesn't contain ``__init__.py`` directly, instead, you have +to use ``find_namespace:``: + +.. code-block:: ini + + [options] + package_dir = + =src + packages = find_namespace: -Another option is to use the "src" layout, where all package code is placed in -the ``src`` directory, like so:: + [options.packages.find_namespace] + where = src +When you install the zipped distribution, ``timmins.foo`` would become +available to your interpreter. - ├── setup.py - ├── src - │   └── namespace - │   └── mypackage - │   ├── __init__.py - │   └── mod1.py - └── tests - └── test_mod1.py +You can think of ``find_namespace:`` as identical to ``find:`` except it +would count a directory as a package even if it doesn't contain ``__init__.py`` +file directly. As a result, this creates an interesting side effect. If you +organize your package like this: -With this layout, the package directory is specified as ``src``, as such:: +.. code-block:: bash - setup(name="namespace.mypackage", - version="0.1", - package_dir={"": "src"}, - packages=find_namespace_packages(where="src")) + foo/ + timmins/ + foo/__init__.py + setup.cfg # or setup.py + tests/ + test_foo/__init__.py -.. _PEP 420: https://www.python.org/dev/peps/pep-0420/ +a naive ``find_namespace:`` would include tests as part of your package to +be installed. A simple way to fix it is to adopt the aforementioned +``src`` layout. -Namespace Packages ------------------- +Legacy Namespace Packages +========================== +The fact you can create namespace package so effortlessly above is credited +to `PEP 420 `_. In the past, it +is more cumbersome to accomplish the same result. -Sometimes, a large package is more useful if distributed as a collection of -smaller eggs. However, Python does not normally allow the contents of a -package to be retrieved from more than one location. "Namespace packages" -are a solution for this problem. When you declare a package to be a namespace -package, it means that the package has no meaningful contents in its -``__init__.py``, and that it is merely a container for modules and subpackages. +Starting with the same layout, there are two pieces you need to add to it. +First, an ``__init__.py`` file directly under your namespace package +directory that contains the following: -The ``pkg_resources`` runtime will then automatically ensure that the contents -of namespace packages that are spread over multiple eggs or directories are -combined into a single "virtual" package. +.. code-block:: python -The ``namespace_packages`` argument to ``setup()`` lets you declare your -project's namespace packages, so that they will be included in your project's -metadata. The argument should list the namespace packages that the egg -participates in. For example, the ZopeInterface project might do this:: + __import__("pkg_resources").declare_namespace(__name__) + +And the ``namespace_packages`` keyword in your ``setup.cfg`` or ``setup.py``: + +.. code-block:: ini + + [options] + namespace_packages = timmins + +.. code-block:: python setup( # ... - namespace_packages=["zope"] + namespace_packages = ['timmins'] ) -because it contains a ``zope.interface`` package that lives in the ``zope`` -namespace package. Similarly, a project for a standalone ``zope.publisher`` -would also declare the ``zope`` namespace package. When these projects are -installed and used, Python will see them both as part of a "virtual" ``zope`` -package, even though they will be installed in different locations. - -Namespace packages don't have to be top-level packages. For example, Zope 3's -``zope.app`` package is a namespace package, and in the future PEAK's -``peak.util`` package will be too. +And your directory should look like this:: -Note, by the way, that your project's source tree must include the namespace -packages' ``__init__.py`` files (and the ``__init__.py`` of any parent -packages), in a normal Python package layout. These ``__init__.py`` files -*must* contain the line:: - - __import__("pkg_resources").declare_namespace(__name__) + /foo/ + src/ + timmins/ + __init__.py + foo/__init__.py + setup.cfg #or setup.py -This code ensures that the namespace package machinery is operating and that -the current package is registered as a namespace package. - -You must NOT include any other code and data in a namespace package's -``__init__.py``. Even though it may appear to work during development, or when -projects are installed as ``.egg`` files, it will not work when the projects -are installed using "system" packaging tools -- in such cases the -``__init__.py`` files will not be installed, let alone executed. - -You must include the ``declare_namespace()`` line in the ``__init__.py`` of -*every* project that has contents for the namespace package in question, in -order to ensure that the namespace will be declared regardless of which -project's copy of ``__init__.py`` is loaded first. If the first loaded -``__init__.py`` doesn't declare it, it will never *be* declared, because no -other copies will ever be loaded! +Repeat the same for other packages and you can achieve the same result as +the previous section. -- cgit v1.2.3 From e53fe0d2eafa85c3ad85c837953f67f4aba28ed3 Mon Sep 17 00:00:00 2001 From: alvyjudy Date: Tue, 26 May 2020 23:47:49 -0400 Subject: docs: cover pkgutil style namespace pkg --- docs/userguide/package_discovery.txt | 41 +++++++++++++++++++++++++++++------- 1 file changed, 33 insertions(+), 8 deletions(-) diff --git a/docs/userguide/package_discovery.txt b/docs/userguide/package_discovery.txt index 77d61770..350a02ad 100644 --- a/docs/userguide/package_discovery.txt +++ b/docs/userguide/package_discovery.txt @@ -22,6 +22,7 @@ included manually in the following manner: .. code-block:: ini [options] + #... packages = mypkg1 mypkg2 @@ -29,6 +30,7 @@ included manually in the following manner: .. code-block:: python setup( + #... packages = ['mypkg1', 'mypkg2'] ) @@ -166,14 +168,24 @@ be installed. A simple way to fix it is to adopt the aforementioned Legacy Namespace Packages -========================== +========================= The fact you can create namespace package so effortlessly above is credited -to `PEP 420 `_. In the past, it -is more cumbersome to accomplish the same result. - -Starting with the same layout, there are two pieces you need to add to it. -First, an ``__init__.py`` file directly under your namespace package -directory that contains the following: +to `PEP 420 `_. It use to be more +cumbersome to accomplish the same result. Historically, there were two methods +to create namespace packages. One is the ``pkg_resources`` style supported by +``setuptools`` and the other one being ``pkgutils`` style offered by +``pkgutils`` module in Python. Both are now considered deprecated despite the +fact they still linger in many existing packages. These two differ in many +subtle yet significant aspects and you can find out more on `Python packaging +user guide `_ + + +``pkg_resource`` style namespace package +---------------------------------------- +This is the method ``setuptools`` directly supports. Starting with the same +layout, there are two pieces you need to add to it. First, an ``__init__.py`` +file directly under your namespace package directory that contains the +following: .. code-block:: python @@ -193,7 +205,9 @@ And the ``namespace_packages`` keyword in your ``setup.cfg`` or ``setup.py``: namespace_packages = ['timmins'] ) -And your directory should look like this:: +And your directory should look like this + +.. code-block:: bash /foo/ src/ @@ -204,3 +218,14 @@ And your directory should look like this:: Repeat the same for other packages and you can achieve the same result as the previous section. + +``pkgutil`` style namespace package +----------------------------------- +This method is almost identical to the ``pkg_resource`` except that the +``__init__.py`` file contains the following: + +.. code-block:: python + + __path__ = __import__('pkgutil').extend_path(__path__, __name__) + +The project layout remains the same and ``setup.cfg`` remains the same. -- cgit v1.2.3 From 70c8c0f39cf737d5eff70e36af5c66973ae1340a Mon Sep 17 00:00:00 2001 From: "Jason R. Coombs" Date: Wed, 27 May 2020 08:54:37 -0400 Subject: Mention that `namespace_packages` is omitted. --- docs/userguide/package_discovery.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/userguide/package_discovery.txt b/docs/userguide/package_discovery.txt index 350a02ad..0e0d27c5 100644 --- a/docs/userguide/package_discovery.txt +++ b/docs/userguide/package_discovery.txt @@ -222,7 +222,8 @@ the previous section. ``pkgutil`` style namespace package ----------------------------------- This method is almost identical to the ``pkg_resource`` except that the -``__init__.py`` file contains the following: +``namespace_packages`` declaration is omitted and the ``__init__.py`` +file contains the following: .. code-block:: python -- cgit v1.2.3