The CFG API for Python¶

If you haven’t used CFG from Python before, you can skip straight to Getting Started with CFG in Python.

Backwards-Incompatible Changes¶

The original implementation was in 2008. The latest version of that implementation is 0.4.2, released in May 2019. A new implementation was started in 2018 (with some changes to the format) and this differs from the earlier implementation in a number of ways:

Format¶

The format now uses true, false and null rather than True, False and None. This is for JSON compatibility.
The format now uses ${A.B.C} for references rather than $A.B.C. This is to allow better expressivity in paths, and to allow interpolation functionality to be implemented.
Multiple strings following one another are concatenated into a single string.

Code¶

There is no support for writing configurations through the API, only for reading them.
config is now a package rather than a module.
The classes ConfigInputStream, ConfigOutputStream, ConfigList, ConfigMerger, ConfigReader, Container, Expression, Mapping, Namespace, Reference, SeqIter and Sequence are not in the new implementation.
The ConfigResolutionError exception is not in the new implementation.
The Config class in the new implementation is completely different.
The functions defaultMergeResolve, defaultStreamOpener, isWord, makePath and overwriteMergeResolve are not in the new implementation.

If your code relies on specific features of the old implementation, be sure to specify config<0.5 in your dependencies.

Modules¶

The Python implementation is divided into three modules:

config contains the high-level API which you will normally interact with.
config.tokens contains code pertaining to tokens and lexical scanning of CFG.
config.parser contains code pertaining to parsing CFG and returning Abstract Syntax Trees (ASTs).

Installation¶

You can install this updated version of the library using

$ pip install config >= 0.5.0

You should install into a virtual environment.

There’s a minimal example of a program that uses CFG here.

Getting Started with CFG in Python¶

A configuration is represented by an instance of the Config class. The constructor for this class can be passed a filename or a stream which contains the text for the configuration. The text is read in, parsed and converted to an object that you can then query. A simple example:

test0.cfg¶

a: 'Hello, '
b: 'world!'
c: {
  d: 'e'
}
'f.g': 'h'
christmas_morning: `2019-12-25 08:39:49`
output: `sys:stdout`
error: `sys.stderr`
home: `$HOME`
module: `logging.handlers:`

Loading a configuration¶

The configuration above can be loaded like this:

>>> import io, os, sys, config
>>> cfg = config.Config('test0.cfg')

This will assume the file is encoded using UTF-8. If that’s not the case, you can pass an encoding= keyword parameter with the desired encoding.

Access elements with keys¶

Accessing elements of the configuration with a simple key is just like using a dictionary:

>>> cfg['a']
'Hello, '
>>> cfg['b']
'world!'

Access elements with paths¶

As well as simple keys, elements can also be accessed using path strings:

>>> cfg['c.d']
'e'

Here, the desired value is obtained in a single step, by (under the hood) walking the path c.d – first getting the mapping at key c, and then the value at d in the resulting mapping.

Note that you can have simple keys which look like paths:

>>> cfg['f.g']
'h'

If a key is given that exists in the configuration, it is used as such, and if it is not present in the configuration, an attempt is made to interpret it as a path. Thus, f.g is present and accessed via key, whereas c.d is not an existing key, so is interpreted as a path.

Access to `datetime` objects¶

You can also get native Python datetime objects from a configuration, by using an ISO date/time pattern in a backtick-string:

>>> cfg['christmas_morning']  #  using `2019-12-25 08:39:49`
datetime.datetime(2019, 12, 25, 8, 39, 49)

Access to other Python objects¶

Access to other Python objects is also possible using the backtick-string syntax, provided that they are either environment values or objects contained within importable modules:

>>> cfg['error']  # using `sys.stderr`
<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>
>>> cfg['error'] is sys.stderr  # Is it the exact same object?
True
>>> cfg['output']  # using `sys:stdout`
<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
>>> cfg['output'] is sys.stdout  # Is it the exact same object?
True

Note

The `sys.stderr` form is only for backward compatibility. You should use the `sys:stderr` form, which will be faster – the all-dots form will try repeated imports to see where the split is between the module and some object within it.

Note that modules themselves can be accessed, just like any other object:

>>> cfg['module']  # using `logging.handlers:`
<module 'logging.handlers' from '/usr/lib/python3.7/logging/handlers.py'>

Access to environment variables¶

To access an environment variable, use a backtick-string of the form `$VARNAME`:

>>> cfg['home'] == os.path.expanduser('~')  # using `$HOME`
True

You can specify a default value to be used if an environment variable isn’t present using the `$VARNAME|default-value` form. Whatever string follows the pipe character (including the empty string) is returned if VARNAME is not a variable in the environment.

Access to computed values¶

Sometimes, it’s useful to have values computed declaratively in the configuration, rather than imperatively in the code that processes the configuration. For example, an overall time period may be specified and other configuration values are fractions thereof. It may also be desirable to perform other simple calculations declaratively, e.g. concatenation of numerous file names to a base directory to get a final pathname.

test0a.cfg¶

total_period : 100
header_time: 0.3 * ${total_period}
steady_time: 0.5 * ${total_period}
trailer_time: 0.2 * ${total_period}
base_prefix: '/my/app/'
log_file: ${base_prefix} + 'test.log'

When this file is read in, the computed values can be accessed directly:

>>> cfg = config.Config('test0a.cfg')
...
>>> cfg['header_time']
30.0
>>> cfg['steady_time']
50.0
>>> cfg['trailer_time']
20.0
>>> cfg['log_file']
'/my/app/test.log'

Including one configuration inside another¶

There are times when it’s useful to include one configuration inside another. For example, consider the following configuration files:

logging.cfg¶

version: 1,
disable_existing_loggers: false,
formatters: {
  brief: {
    class: 'logging.Formatter',
    format: '%(name)20.20s %(message)s'
  }
},
handlers: {
  file: {
    level: 'INFO',
    class: `logging:FileHandler`,
    formatter: 'brief',
    filename: 'run/server.log',
    mode: 'w',
    encoding: 'utf-8'
  },
  error: {
    level: 'ERROR',
    class: `logging:FileHandler`,
    formatter: 'brief',
    filename: 'run/server-errors.log',
    mode: 'w',
    encoding: 'utf-8'
  },
  debug: {
    level: 'DEBUG',
    class: `logging:FileHandler`,
    formatter: 'brief',
    filename: 'run/server-debug.log',
    mode: 'w',
    encoding: 'utf-8'
  }
},
loggers: {
  mylib: {
    level: 'INFO'
  }
  'mylib.detail': {
    level: 'DEBUG'
  }
},
root: {
  handlers: ['file', 'error', 'debug'],
  level: 'WARNING'
}

redirects.cfg¶

cookies: {
  url: 'http://www.allaboutcookies.org/',
  permanent: false
},
freeotp: {
  url: 'https://freeotp.github.io/',
  permanent: false
},
'google-auth': {
  url: 'https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2',
  permanent: false
}

main.cfg¶

secret: 'some random application secret',
port: 8000,
server: 'cheroot',
sitename: 'My Test Site',
default_access: 'public',
ignore_trailing_slashes: true,
site_options: {
  want_ipinfo: false,
  show_form: true,
  cookie_bar: true
},
connection: 'postgres+pool://db_user:db_pwd@localhost:5432/db_name',
debug: true,
captcha_length: 4,
captcha_timeout: 5,
session_timeout: 7 * 24 * 60 * 60,  # 7 days in seconds
redirects: @'redirects.cfg',
email: {
  sender: 'no-reply@my-domain.com',
  host: 'smtp.my-domain.com:587',
  user: 'smtp-user',
  password: 'smtp-pwd'
}
logging: @'logging.cfg'

The main.cfg contents have been kept to the highest-level values, within logging and redirection configuration relegated to other files logging.cfg and redirects.cfg which are then included in main.cfg. This allows the high-level configuration to be more readable at a glance, and even allows the separate configuration files to be e.g. maintained by different people.

The contents of the sub-configurations are easily accessible from the main configuration just as if they had been defined in the same file:

>>> import config
>>> cfg = config.Config('main.cfg')
...
>>> cfg['redirects.freeotp.url']
'https://freeotp.github.io/'
>>> cfg['redirects.freeotp.permanent']
False
>>> cfg['logging.root.level']
'WARNING'
>>> cfg['logging.handlers.file.class']
<class 'logging.FileHandler'>

Avoiding unnecessary repetition¶

Don’t Repeat Yourself (DRY) is a useful principle to follow. CFG can help with this. You may have noticed that the logging.cfg file above has some repetitive elements:

logging.cfg (partial)¶

handlers: {
  file: {
    level: 'INFO',
    class: `logging:FileHandler`,
    formatter: 'brief',
    filename: 'run/server.log',
    mode: 'w',
    encoding: 'utf-8'
  },
  error: {
    level: 'ERROR',
    class: `logging:FileHandler`,
    formatter: 'brief',
    filename: 'run/server-errors.log',
    mode: 'w',
    encoding: 'utf-8'
  },
  debug: {
    level: 'DEBUG',
    class: `logging:FileHandler`,
    formatter: 'brief',
    filename: 'run/server-debug.log',
    mode: 'w',
    encoding: 'utf-8'
  }
},

This portion could be rewritten as:

logging.cfg (partial)¶

defs: {
  base_file_handler: {
    class: `logging:FileHandler`,
    formatter: 'brief',
    mode: 'w',
    encoding: 'utf-8'
  }
},
handlers: {
  file: ${defs.base_file_handler} + {
    level: 'INFO',
    filename: 'run/server.log',
    mode: 'a',
  },
  error: ${defs.base_file_handler} + {
    level: 'ERROR',
    filename: 'run/server-errors.log',
  },
  debug: ${defs.base_file_handler} + {
    level: 'DEBUG',
    filename: 'run/server-debug.log',
  }
},

where the common elements are separated out and just referenced where they are needed. We find it useful to put all things which will be reused like this in one place in the condiguration, so we always know where to go to make changes. The key used is conventionally defs or base, though it can be anything you like.

Access is just as before, and provides the same results:

>>> import config
>>> cfg = config.Config('main.cfg')
...
>>> cfg['logging.handlers.file.class']
<class 'logging.FileHandler'>
>>> cfg['logging.handlers.file.level']
'INFO'
>>> cfg['logging.handlers.file.formatter']
'brief'
>>> cfg['logging.handlers.file.encoding']
'utf-8'
>>> cfg['logging.handlers.file.mode']
'a'
>>> cfg['logging.handlers.file.filename']
'run/server.log'
>>> cfg['logging.handlers.error.mode']
'w'
>>> cfg['logging.handlers.error.filename']
'run/server-errors.log'

The definition of logging.handlers.file as ${defs.base_file_handler} + { level: 'INFO', filename: 'run/server.log', mode: 'a' } has resulted in an evaluation which first fetches the defs.base_file_handler value, which is a mapping, and “adds” to that the literal mapping which defines the level, filename and mode keys. The + operation for mappings is implemented as a copy of the left-hand side merged with the right-hand side. Note that the mode value for logging.handlers.file is overridden by the right-hand side to 'a', whereas that for e.g. logging.handlers.error is unchanged as 'w'.

We could do some further refinement by factoring out the common location for the log files:

logging.cfg (partial)¶

defs: {
  base_file_handler: {
    class: `logging:FileHandler`,
    formatter: 'brief',
    mode: 'w',
    encoding: 'utf-8'
  }
  log_prefix: 'run/',
},
handlers: {
  file: ${defs.base_file_handler} + {
    level: 'INFO',
    filename: ${defs.log_prefix} + 'server.log',
    mode: 'a',
  },
  error: ${defs.base_file_handler} + {
    level: 'ERROR',
    filename: ${defs.log_prefix} + 'server-errors.log',
  },
  debug: ${defs.base_file_handler} + {
    level: 'DEBUG',
    filename: ${defs.log_prefix} + 'server-debug.log',
  }
}

with the same result as before. It is slightly more verbose than before, but the location of all files can be changed in just one place now, as opposed to three, as it was before.

A larger example - Django configuration¶

Let’s look at a slightly larger example:

log1.cfg¶

version: 1
disable_existing_loggers: false
formatters: {
    verbose: {
        format: '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
        style: '{'
    },
    simple: {
        format: '{levelname} {message}',
        style: '{',
    },
},
filters: {
    special: {
        '()': `project.logging:SpecialFilter`,
        foo: 'bar',
    },
    require_debug_true: {
        '()': `django.utils.log:RequireDebugTrue`,
    },
},
handlers: {
    console: {
        level: 'INFO',
        filters: ['require_debug_true'],
        class: 'logging.StreamHandler',
        formatter: 'simple'
    },
    mail_admins: {
        level: 'ERROR',
        class: 'django.utils.log.AdminEmailHandler',
        filters: ['special']
    }
},
loggers: {
    django: {
        handlers: ['console'],
        propagate: true,
    },
    'django.request': {
        handlers: ['mail_admins'],
        level: 'ERROR',
        propagate: false,
    },
    'myproject.custom': {
        handlers: ['console', 'mail_admins'],
        level: 'INFO',
        filters: ['special']
    }
}

This is the analogue of the configuration dictionary for the “fairly complex logging setup” documented in Django 2.2.

Let’s assume we have both Django and the following on Python’s sys.path:

project/logging.py¶

import logging

class SpecialFilter(logging.Filter):
    def __init__(self, foo):
        self.foo = foo

    def filter(self, record):
        return True

We can load the configuration easily enough:

>>> import config, logging.config
>>> cfg = config.Config('log1.cfg')
...
>>> logging.config.dictConfig(cfg.as_dict())
>>> logging.getLogger('django').handlers
[<StreamHandler <stderr> (INFO)>]
>>> logging.getLogger('django.request').handlers
[<AdminEmailHandler (ERROR)>]
>>> logging.getLogger('myproject.custom').handlers
[<StreamHandler <stderr> (INFO)>, <AdminEmailHandler (ERROR)>]

Classes¶

The `Location` class¶

This represents a source location and has two integer attributes, line and column. The line must be positive and the column non-negative (newlines have an ending column of zero, as column 1 would have the first character of the next line).

The `ConfigError` class¶

This is an Exception subclass which has an error message in the message attribute and a Location instance in the location attribute indicating the position in the source of the error, where appropriate. (Some errors don’t have a location, and in such cases the location attribute of the instance will be set to None.)

The `Config` class¶

This class implements access to a CFG configuration. You’ll generally interface to CFG files using this class. When you pass in a stream or file path to the constructor the CFG source in the stream or file is parsed and converted into an internal form which can then be accessed in a manner analogous to a Python dictionary.

class Config¶

Attributes

path¶: The path to the file from which the configuration has been read. You won’t usually need to set this, unless you want to load a configuration from text that references included files. In that case, set the path to a value whose parent directory contains the included files, or specify relevant directories using include_path. When a configuration is loaded from a file, this attribute is automatically set.

include_path¶: A list of directories which is searched for included sub-configurations if the parent directory of path (or the current directory, if path isn’t set) doesn’t contain them.

context¶: A mapping of strings to objects which is used whenever an identifier is encountered when evaluating an expression in a configuration file.

no_duplicates¶: Whether keys are allowed to be duplicated in mappings. This defaults to True if not provided – a ConfigError is raised if a duplicate key is seen. If False, then if a duplicate key is seen, its value silently replaces the value associated with the earlier appearance of the same key.

strict_conversions¶: If True, ConfigError is raised if a backtick-string can’t be converted. This defaults to True if not provided. It’s intended to help catch typos in backtick-strings.

Methods

__init__(stream_or_path, **kwargs)

Returns a new instance.

Parameters:

Parameters:	stream_or_path – Either a stream containing the CFG source to be read (which could be an instance of `io.StringIO` or a disk file opened in text mode with an appropriate encoding), or the path to a file containing CFG source. It can also be `None`, in which case a stream needs to be passed to the `load()` method or a file path to the `load_file()` method in order to load a configuration into the instance. kwargs – Keyword parameters, see below.
Keyword Parameters:	cache – If set to `True`, a cache dictionary is initialised and used to cache retrieved values. If not provided, it defaults to `False`. context – A dictionary which is used to look up variables. If not provided, an empty dictionary is used. include_path – A list of directories to be searched for included configurations when an include expression is seen. If not provided, an empty list is used. Note that the parent directory of the `path` attribute (or the current directory if that isn’t set) is always searched for included configurations, before the `include_path` is consulted. no_duplicates – Whether keys are allowed to be duplicated in mappings. This defaults to `True` if not provided – a `ConfigError` is raised if a duplicate key is seen. If `False`, then if a duplicate key is seen, its value silently replaces the value associated with the earlier appearance of the same key. strict_conversions – If `True`, back-tick string conversion raises a `ConfigError` if a string can’t be converted. This defaults to `True` if not provided. It’s intended to help catch typos in backtick-strings. path – If provided, this will be used to set the `path` attribute of the instance. You don’t need to provide this if stream_or_path is a path.

stream_or_path – Either a stream containing the CFG source to be read (which could be an instance of io.StringIO or a disk file opened in text mode with an appropriate encoding), or the path to a file containing CFG source. It can also be None, in which case a stream needs to be passed to the load() method or a file path to the load_file() method in order to load a configuration into the instance.
kwargs – Keyword parameters, see below.

Keyword Parameters:

cache – If set to True, a cache dictionary is initialised and used to cache retrieved values. If not provided, it defaults to False.
context – A dictionary which is used to look up variables. If not provided, an empty dictionary is used.
include_path – A list of directories to be searched for included configurations when an include expression is seen. If not provided, an empty list is used. Note that the parent directory of the path attribute (or the current directory if that isn’t set) is always searched for included configurations, before the include_path is consulted.
no_duplicates – Whether keys are allowed to be duplicated in mappings. This defaults to True if not provided – a ConfigError is raised if a duplicate key is seen. If False, then if a duplicate key is seen, its value silently replaces the value associated with the earlier appearance of the same key.
strict_conversions – If True, back-tick string conversion raises a ConfigError if a string can’t be converted. This defaults to True if not provided. It’s intended to help catch typos in backtick-strings.
path – If provided, this will be used to set the path attribute of the instance. You don’t need to provide this if stream_or_path is a path.

load(stream)¶

Loads CFG source into this instance.

Parameters:	stream – A stream containing the CFG source to be read. This could be an instance of `io.StringIO` or a disk file opened in text mode with an appropriate encoding. The source must be for a mapping body or a mapping. All top-level configurations must be of this type, though included sub-configurations could also take the form of a list.

close()¶: Closes this instance, after which no other methods should be called on it. If can_close is True, the stream will be closed.

as_dict()¶: Return’s this instance’s data as a Python dictionary.

get(key, default_value)¶

Parameters:	key (str) – A key or path into this configuration. default_value – A value to return if `key` cannot be found.

Get an element from this instance using key, and returning the corresponding value or default_value if the key isn’t present. The key can either be an actual key or a path.

Note that normal indexed access cfg[key] is also possible, and in this case, if the key or path are not found in the configuration, a ConfigError is raised.

A ConfigError will also be raised if a path is invalid in some way.