The CFG API for Python

If you haven’t used CFG from Python before, you can skip straight to Getting Started with CFG in Python.

Backwards-Incompatible Changes

The original implementation was in 2008. The latest version of that implementation is 0.4.2, released in May 2019. A new implementation was started in 2018 (with some changes to the format) and this differs from the earlier implementation in a number of ways:

Format

  • The format now uses true, false and null rather than True, False and None. This is for JSON compatibility.
  • The format now uses ${A.B.C} for references rather than $A.B.C. This is to allow better expressivity in paths.
  • Multiple strings following one another are concatenated into a single string.

Code

  • There is no support for writing configurations through the API, only for reading them.
  • config is now a package rather than a module.
  • The classes ConfigInputStream, ConfigOutputStream, ConfigList, ConfigMerger, ConfigReader, Container, Expression, Mapping, Namespace, Reference, SeqIter and Sequence are not in the new implementation.
  • The ConfigResolutionError exception is not in the new implementation.
  • The Config class in the new implementation is completely different.
  • The functions defaultMergeResolve, defaultStreamOpener, isWord, makePath and overwriteMergeResolve are not in the new implementation.

If your code relies on specific features of the old implementation, be sure to specify config<0.5 in your dependencies.

Modules

The Python implementation is divided into three modules:

  • config contains the high-level API which you will normally interact with.
  • config.tokens contains code pertaining to tokens and lexical scanning of CFG.
  • config.parser contains code pertaining to parsing CFG and returning Abstract Syntax Trees (ASTs).

Installation

You can install this updated version of the library using

$ pip install config >= 0.5.0

You should install into a virtual environment.

Getting Started with CFG in Python

A configuration is represented by an instance of the Config class. The constructor for this class can be passed a filename or a stream which contains the text for the configuration. The text is read in, parsed and converted to an object that you can then query. A simple example:

test0.cfg
a: 'Hello, '
b: 'world!'
c: {
  d: 'e'
}
'f.g': 'h'
christmas_morning: `2019-12-25 08:39:49`
output: `sys:stdout`
error: `sys.stderr`
home: `$HOME`
module: `logging.handlers:`

Loading a configuration

The configuration above can be loaded like this:

>>> import io, os, sys, config
>>> cfg = config.Config('test0.cfg')

This will assume the file is encoded using UTF-8. If that’s not the case, you can pass an encoding= keyword parameter with the desired encoding.

Access elements with keys

Accessing elements of the configuration with a simple key is just like using a dictionary:

>>> cfg['a']
'Hello, '
>>> cfg['b']
'world!'

Access elements with paths

As well as simple keys, elements can also be accessed using path strings:

>>> cfg['c.d']
'e'

Here, the desired value is obtained in a single step, by (under the hood) walking the path c.d – first getting the mapping at key c, and then the value at d in the resulting mapping.

Note that you can have simple keys which look like paths:

>>> cfg['f.g']
'h'

If a key is given that exists in the configuration, it is used as such, and if it is not present in the configuration, an attempt is made to interpret it as a path. Thus, f.g is present and accessed via key, whereas c.d is not an existing key, so is interpreted as a path.

Access to datetime objects

You can also get native Python datetime objects from a configuration, by using an ISO date/time pattern in a backtick-string:

>>> cfg['christmas_morning']  #  using `2019-12-25 08:39:49`
datetime.datetime(2019, 12, 25, 8, 39, 49)

Access to other Python objects

Access to other Python objects is also possible using the backtick-string syntax, provided that they are either environment values or objects contained within importable modules:

>>> cfg['error']  # using `sys.stderr`
<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>
>>> cfg['error'] is sys.stderr  # Is it the exact same object?
True
>>> cfg['output']  # using `sys:stdout`
<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
>>> cfg['output'] is sys.stdout  # Is it the exact same object?
True

Note

The `sys.stderr` form is only for backward compatibility. You should use the `sys:stderr` form, which will be faster – the all-dots form will try repeated imports to see where the split is between the module and some object within it.

Note that modules themselves can be accessed, just like any other object:

>>> cfg['module']  # using `logging.handlers:`
<module 'logging.handlers' from '/usr/lib/python3.7/logging/handlers.py'>

Access to environment variables

To access an environment variable, use a backtick-string of the form `$VARNAME`:

>>> cfg['home'] == os.path.expanduser('~')  # using `$HOME`
True

You can specify a default value to be used if an environment variable isn’t present using the `$VARNAME|default-value` form. Whatever string follows the pipe character (including the empty string) is returned if VARNAME is not a variable in the environment.

Access to computed values

Sometimes, it’s useful to have values computed declaratively in the configuration, rather than imperatively in the code that processes the configuration. For example, an overall time period may be specified and other configuration values are fractions thereof. It may also be desirable to perform other simple calculations declaratively, e.g. concatenation of numerous file names to a base directory to get a final pathname.

test0a.cfg
total_period : 100
header_time: 0.3 * ${total_period}
steady_time: 0.5 * ${total_period}
trailer_time: 0.2 * ${total_period}
base_prefix: '/my/app/'
log_file: ${base_prefix} + 'test.log'

When this file is read in, the computed values can be accessed directly:

>>> cfg = config.Config('test0a.cfg')
...
>>> cfg['header_time']
30.0
>>> cfg['steady_time']
50.0
>>> cfg['trailer_time']
20.0
>>> cfg['log_file']
'/my/app/test.log'

Including one configuration inside another

There are times when it’s useful to include one configuration inside another. For example, consider the following configuration files:

logging.cfg
version: 1,
disable_existing_loggers: false,
formatters: {
  brief: {
    class: 'logging.Formatter',
    format: '%(name)20.20s %(message)s'
  }
},
handlers: {
  file: {
    level: 'INFO',
    class: `logging:FileHandler`,
    formatter: 'brief',
    filename: 'run/server.log',
    mode: 'w',
    encoding: 'utf-8'
  },
  error: {
    level: 'ERROR',
    class: `logging:FileHandler`,
    formatter: 'brief',
    filename: 'run/server-errors.log',
    mode: 'w',
    encoding: 'utf-8'
  },
  debug: {
    level: 'DEBUG',
    class: `logging:FileHandler`,
    formatter: 'brief',
    filename: 'run/server-debug.log',
    mode: 'w',
    encoding: 'utf-8'
  }
},
loggers: {
  mylib: {
    level: 'INFO'
  }
  'mylib.detail': {
    level: 'DEBUG'
  }
},
root: {
  handlers: ['file', 'error', 'debug'],
  level: 'WARNING'
}
redirects.cfg
cookies: {
  url: 'http://www.allaboutcookies.org/',
  permanent: false
},
freeotp: {
  url: 'https://freeotp.github.io/',
  permanent: false
},
'google-auth': {
  url: 'https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2',
  permanent: false
}
main.cfg
secret: 'some random application secret',
port: 8000,
server: 'cheroot',
sitename: 'My Test Site',
default_access: 'public',
ignore_trailing_slashes: true,
site_options: {
  want_ipinfo: false,
  show_form: true,
  cookie_bar: true
},
connection: 'postgres+pool://db_user:db_pwd@localhost:5432/db_name',
debug: true,
captcha_length: 4,
captcha_timeout: 5,
session_timeout: 7 * 24 * 60 * 60,  # 7 days in seconds
redirects: @'redirects.cfg',
email: {
  sender: 'no-reply@my-domain.com',
  host: 'smtp.my-domain.com:587',
  user: 'smtp-user',
  password: 'smtp-pwd'
}
logging: @'logging.cfg'

The main.cfg contents have been kept to the highest-level values, within logging and redirection configuration relegated to other files logging.cfg and redirects.cfg which are then included in main.cfg. This allows the high-level configuration to be more readable at a glance, and even allows the separate configuration files to be e.g. maintained by different people.

The contents of the sub-configurations are easily accessible from the main configuration just as if they had been defined in the same file:

>>> import config
>>> cfg = config.Config('main.cfg')
...
>>> cfg['redirects.freeotp.url']
'https://freeotp.github.io/'
>>> cfg['redirects.freeotp.permanent']
False
>>> cfg['logging.root.level']
'WARNING'
>>> cfg['logging.handlers.file.class']
<class 'logging.FileHandler'>

Avoiding unnecessary repetition

Don’t Repeat Yourself (DRY) is a useful principle to follow. CFG can help with this. You may have noticed that the logging.cfg file above has some repetitive elements:

logging.cfg (partial)
handlers: {
  file: {
    level: 'INFO',
    class: `logging:FileHandler`,
    formatter: 'brief',
    filename: 'run/server.log',
    mode: 'w',
    encoding: 'utf-8'
  },
  error: {
    level: 'ERROR',
    class: `logging:FileHandler`,
    formatter: 'brief',
    filename: 'run/server-errors.log',
    mode: 'w',
    encoding: 'utf-8'
  },
  debug: {
    level: 'DEBUG',
    class: `logging:FileHandler`,
    formatter: 'brief',
    filename: 'run/server-debug.log',
    mode: 'w',
    encoding: 'utf-8'
  }
},

This portion could be rewritten as:

logging.cfg (partial)
defs: {
  base_file_handler: {
    class: `logging:FileHandler`,
    formatter: 'brief',
    mode: 'w',
    encoding: 'utf-8'
  }
},
handlers: {
  file: ${defs.base_file_handler} + {
    level: 'INFO',
    filename: 'run/server.log',
    mode: 'a',
  },
  error: ${defs.base_file_handler} + {
    level: 'ERROR',
    filename: 'run/server-errors.log',
  },
  debug: ${defs.base_file_handler} + {
    level: 'DEBUG',
    filename: 'run/server-debug.log',
  }
},

where the common elements are separated out and just referenced where they are needed. We find it useful to put all things which will be reused like this in one place in the condiguration, so we always know where to go to make changes. The key used is conventionally defs or base, though it can be anything you like.

Access is just as before, and provides the same results:

>>> import config
>>> cfg = config.Config('main.cfg')
...
>>> cfg['logging.handlers.file.class']
<class 'logging.FileHandler'>
>>> cfg['logging.handlers.file.level']
'INFO'
>>> cfg['logging.handlers.file.formatter']
'brief'
>>> cfg['logging.handlers.file.encoding']
'utf-8'
>>> cfg['logging.handlers.file.mode']
'a'
>>> cfg['logging.handlers.file.filename']
'run/server.log'
>>> cfg['logging.handlers.error.mode']
'w'
>>> cfg['logging.handlers.error.filename']
'run/server-errors.log'

The definition of logging.handlers.file as ${defs.base_file_handler} + { level: 'INFO', filename: 'run/server.log', mode: 'a' } has resulted in an evaluation which first fetches the defs.base_file_handler value, which is a mapping, and “adds” to that the literal mapping which defines the level, filename and mode keys. The + operation for mappings is implemented as a copy of the left-hand side merged with the right-hand side. Note that the mode value for logging.handlers.file is overridden by the right-hand side to 'a', whereas that for e.g. logging.handlers.error is unchanged as 'w'.

We could do some further refinement by factoring out the common location for the log files:

logging.cfg (partial)
defs: {
  base_file_handler: {
    class: `logging:FileHandler`,
    formatter: 'brief',
    mode: 'w',
    encoding: 'utf-8'
  }
  log_prefix: 'run/',
},
handlers: {
  file: ${defs.base_file_handler} + {
    level: 'INFO',
    filename: ${defs.log_prefix} + 'server.log',
    mode: 'a',
  },
  error: ${defs.base_file_handler} + {
    level: 'ERROR',
    filename: ${defs.log_prefix} + 'server-errors.log',
  },
  debug: ${defs.base_file_handler} + {
    level: 'DEBUG',
    filename: ${defs.log_prefix} + 'server-debug.log',
  }
}

with the same result as before. It is slightly more verbose than before, but the location of all files can be changed in just one place now, as opposed to three, as it was before.

A larger example - Django configuration

Let’s look at a slightly larger example:

log1.cfg
version: 1
disable_existing_loggers: false
formatters: {
    verbose: {
        format: '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
        style: '{'
    },
    simple: {
        format: '{levelname} {message}',
        style: '{',
    },
},
filters: {
    special: {
        '()': `project.logging:SpecialFilter`,
        foo: 'bar',
    },
    require_debug_true: {
        '()': `django.utils.log:RequireDebugTrue`,
    },
},
handlers: {
    console: {
        level: 'INFO',
        filters: ['require_debug_true'],
        class: 'logging.StreamHandler',
        formatter: 'simple'
    },
    mail_admins: {
        level: 'ERROR',
        class: 'django.utils.log.AdminEmailHandler',
        filters: ['special']
    }
},
loggers: {
    django: {
        handlers: ['console'],
        propagate: true,
    },
    'django.request': {
        handlers: ['mail_admins'],
        level: 'ERROR',
        propagate: false,
    },
    'myproject.custom': {
        handlers: ['console', 'mail_admins'],
        level: 'INFO',
        filters: ['special']
    }
}

This is the analogue of the configuration dictionary for the “fairly complex logging setup” documented in Django 2.2.

Let’s assume we have both Django and the following on Python’s sys.path:

project/logging.py
import logging

class SpecialFilter(logging.Filter):
    def __init__(self, foo):
        self.foo = foo

    def filter(self, record):
        return True

We can load the configuration easily enough:

>>> import config, logging.config
>>> cfg = config.Config('log1.cfg')
...
>>> logging.config.dictConfig(cfg.as_dict())
>>> logging.getLogger('django').handlers
[<StreamHandler <stderr> (INFO)>]
>>> logging.getLogger('django.request').handlers
[<AdminEmailHandler (ERROR)>]
>>> logging.getLogger('myproject.custom').handlers
[<StreamHandler <stderr> (INFO)>, <AdminEmailHandler (ERROR)>]

Classes

The Config class

This class implements access to a CFG configuration. You’ll generally interface to CFG files using this class. When you pass in a stream or file path to the constructor the CFG source in the stream or file is parsed and converted into an internal form which can then be accessed in a manner analogous to a Python dictionary.

class Config

Attributes

path

The path to the file from which the configuration has been read.

rootdir

The directory of path, which is used to search for included sub-configurations before scanning include_path.

include_path

A list of directories which is searched for included sub-configurations if rootdir doesn’t contain them.

context

A mapping of strings to objects which is used whenever an identifier is encountered when evaluating an expression in a configuration file.

no_duplicates

Whether keys are allowed to be duplicated in mappings. This defaults to True if not provided – a ConfigError is raised if a duplicate key is seen. If False, then if a duplicate key is seen, its value silently replaces the value associated with the earlier appearance of the same key.

strict_conversions

If True, convert_string() raises a ConfigError if a string can’t be converted. This defaults to True if not provided. It’s intended to help catch typos in backtick-strings.

Methods

__init__(stream_or_path, **kwargs)

Returns a new instance.

Parameters:
  • stream_or_path – Either a stream containing the CFG source to be read (which could be an instance of io.StringIO or a disk file opened in text mode with an appropriate encoding), or the path to a file containing CFG source. It can also be None, in which case a stream needs to be passed to the load() method or a file path to the load_file() method in order to load a configuration into the instance.
  • kwargs – Keyword parameters, see below.
Keyword Parameters:
  • cache – If set to True, a cache dictionary is initialised and used to cache retrieved values. If not provided, it defaults to False.
  • context – A dictionary which is used to look up variables. If not provided, an empty dictionary is used.
  • include_path – A list of directories to be searched for included configurations when an include expression is seen. If not provided, an empty list is used. Note that the rootdir attribute is always searched for included configurations, so the include_path is only needed if the included configuration file is not in the same place as the including configuration file.
  • no_duplicates – Whether keys are allowed to be duplicated in mappings. This defaults to True if not provided – a ConfigError is raised if a duplicate key is seen. If False, then if a duplicate key is seen, its value silently replaces the value associated with the earlier appearance of the same key.
  • strict_conversions – If True, convert_string() raises a ConfigError if a string can’t be converted. This defaults to True if not provided. It’s intended to help catch typos in backtick-strings.
  • path – If provided, this will be used to set the path and rootdir attributes of the instance. You don’t need to provide this if stream_or_path is a path.
  • rootdir – If provided, this should be the directory of the absolute path where the configuration is logically located. If not provided, then if stream_or_path is an open disk file, or it’s a path, the value is used to determine the root_dir value. Otherwise, the current directory is used. Included configurations are always looked for here, before the include_path is searched.
load(stream)

Loads CFG source into this instance.

Parameters:stream – A stream containing the CFG source to be read. This could be an instance of io.StringIO or a disk file opened in text mode with an appropriate encoding.

The source must be for a mapping body or a mapping. All top-level configurations must be of this type, though included sub-configurations could also take the form of a list.

close()

Closes this instance, after which no other methods should be called on it. If can_close is True, the stream will be closed.

as_dict()

Return’s this instance’s data as a Python dictionary.

convert_string(s)

Convert the backtick-string s into an object. If it can’t be converted and strict_conversions is true, a ConfigError will be raised.