History¶

What, another new configuration format?¶

It’s not a new format. Back in 2008, we announced a configuration format which had to meet a number of requirements (click on the little arrows below to see more detail):

Support a nested hierarchy of elements
Support a nested hierarchy of elements: the level of nesting of configuration elements should not be arbitrarily restricted. This requirement rules out .ini-like formats, which typically allow no more than two levels of nesting. However, a JSON-like format is a natural fit for this requirement. In this scheme, a configuration consists of key-value mappings, lists and scalar values (strings, numbers, Boolean values and so on). A mapping in JSON is delimited with { and }, and a list is delimited with [ and ]. It’s also sometimes useful to consider a mapping body, which is the part of a mapping between { and }, and a list body, which is the part of a list between [ and ].
Comments should be allowed
Comments should be allowed: this is useful to document things in the configuration which aren’t obvious. This requirement rules out standard JSON. Identifiers should be usable as mapping keys: requiring quoted strings, as JSON does, introduces a lot of noise without any real benefit. Of course, strings which aren’t identifiers should be allowed as keys - they just shouldn’t always be required.
Trailing commas should be allowed
Trailing commas should be allowed: lists and maps should allow a single trailing comma, as it makes life easier in numerous ways. This requirement also rules out standard JSON.
Newlines, as well as commas, should be usable to separate elements
Newlines, as well as commas, should be usable to separate elements: sometimes, newlines are essential in making a configuration more readable. If possible, newlines should be usable as element delimiters, as well as commas which are used in JSON and other similar formats.
Ordinary strings can be single-quoted
Ordinary strings can be single-quoted: by ordinary strings, we mean normal strings rather than “raw” strings which don’t support processing of escape sequences such as \n or \t. Such strings should be specifiable using either single- or double-quotes. This allows specification of "O'Flaherty" or '5" floppy' straightforwardly, without the need for escape sequences.
Full Unicode support
Full Unicode support: it should be possible to represent any Unicode text in configurations.
Multi-line strings
Multi-line strings: It should be possible to specify string literals which span multiple lines in a simple and easy way.
It should be possible to use hexadecimal numbers
It should be possible to use hexadecimal numbers: there are situations where hexadecimal numbers (e.g. 0xabcd) confer better readability than decimals. By extension, octal numbers (e.g. 0o377) and binary numbers (e.g. 0b01100111) would also be nice to have.
Syntactically valid JSON should be accepted
Syntactically valid JSON should be accepted: this would ease transition to CFG from JSON configurations, which were/are in common use.
Allow other configurations to be included
Allow other configurations to be included: rather than requiring an entire configuration to be in a single file, it would be advantageous to allow configuration files to somehow include other configuration files. This would allow sharing of common component configurations (e.g. logging) across multiple projects, and also allow users to inspect and work on disparate parts of the configuration independently (to some extent).
Reference one part of a configuration from another
Reference one part of a configuration from another: it should be possible, in the interests of DRY (Don’t-Repeat-Yourself), to allow a part of a configuration to be referenced from another.
Allow access to application state
Allow access to application state: there are situations where it can be useful to pick up configuration values from aspects of the application state, such as environment variables visible to the application, and present them in unified fashion through a single configuration point. This could be extended to other program internals, such as standard process streams (stdin, stdout and stderr, or to allow specification of specific entry points in the program using the configuration).
Allow composition of parts of a configuration from other parts
Allow composition of parts of a configuration from other parts: there are often situations where you want to use part of an existing configuration and put it together in some way with other things. Composition should allow operations such as list composition by concatenation and slicing, and mapping composition by merging mappings. Composition might require simple expression evaluation, but without unconstrained evaluation or the need for Turing-completeness.
Allow a configuration file to specify any of a mapping, a list or a mapping body
Allow a configuration file to specify any of a mapping, a list or a mapping body: this avoids the need for unnecessary boilerplate, while keeping things unambiguous as far as possible.
Order independence apart from lists
Order independence apart from lists: Apart from lists, which are intrinsically ordered, other elements in the configuration should be independent of ordering.
Copy-and-paste and typo protection
Copy-and-paste and typo protection: At least some protection should be provided against common copy-and-paste and typographical errors, such as disallowing duplicate keys in a mapping (which can commonly occur due to typos or copy-and-paste-and-forget-to-change-after-pasting).

Back in 2004 when we first started looking, we couldn’t find anything which met these requirements. So we developed this configuration format and a Python module to make use of it. The module was first published on the Python Package Index in 2004 (but not especially publicised). It’s been used on a few internal projects, and had some users on PyPI, but not seen much development after 2010 - because the module met many, though not all, of the above requirements.

Review of other systems¶

Since 2010, a number of new configuration formats have been proposed: for example, HOCON (2011), JSON5 (2012), TOML (2013), HJSON (2014), and SANE (2018, though it now appears to be inactive), to name but five.

We investigated these new formats to see which might meet our original requirements. All of them support the requirement to support N-level hierarchical configurations, and to allow the full range of Unicode characters in strings. Our findings are summarised in the table below.

Requirement	JSON	JSON5	HJSON	HOCON	SANE	TOML	CFG
Comments allowed?	No	Line, block	Line, block	Line	Line	Line	Line
Allow trailing commas?	No	Yes	Yes	Yes	Yes	Yes	Yes
Newlines (as well as commas) can separate elements?	No	No	Yes	Yes	Yes	Yes	Yes
Identifiers usable as mapping keys?	No	Yes	Yes	Yes	Yes	Yes	Yes
Ordinary strings can be single-quoted?	No	Yes	Yes	No	No	No	Yes
Multi-line strings?	No	Yes	Yes	Yes	Yes	Yes	Yes
Hexadecimal numbers?	No	Yes	No	No	Yes	Yes	Yes
Syntactically valid JSON accepted?	Yes	Yes	Yes	Yes	No	No	Yes
Include other configurations?	No	No	No	Yes	No	No	Yes
Reference parts of a configuration?	No	No	No	Yes	No	No	Yes
Allow access to environment variables?	No	No	No	Yes	No	No	Yes
Allow composing from different parts of a configuration?	No	No	No	Yes	No	No	Yes
Configuration file type	Mapping, List	Mapping	Mapping	Mapping, Mapping body	Mapping body	Mapping body	Mapping, Mapping body, List
Order independence?	Yes	Yes	Yes	No	Yes	No	Yes

As can be seen from the table above, none of the other formats looked at (which are commonly proposed for use in application configuration) meet all the requirements we originally identified. HOCON, which comes closest in terms of number of requirements satisfied or partially satisfied, has numerous perversities which make it seem like it’s not a good fit for our needs: for example, in HOCON the fragment:

3.14 : 42

is interpreted as the key-value pair "3" : { "14" : 42 }, which seems bizarre. Note: TOML has the same behaviour.

We didn’t look at configuration systems that are like programming languages, such as Dhall, CUE or Jsonnet – they are a different kind of thing.

Project Status¶

Following the review, we decided that rather than switching to one of these newer formats, it was worth updating our old implementation to work better with recent Python versions, as well as to provide implementations for other platforms: the JVM (using Kotlin), .NET, Go, Rust, D, JavaScript, Ruby, Elixir, Nim and Dart.

As well as providing useful functionality for application and library configuration in a range of environments, these implementations also serve as an example of comparative development in these various languages of a set of functionality which is non-trivial, but not so large as to be difficult to assimilate / understand.

Reporting Problems¶

If you run into problems, you can raise issues against the issue trackers relating to the relevant implementations using the links below. These are hosted in the relevant source repositories for these implementations.

History