History

What, another new configuration format?

It’s not a new format. Back in 2008, we were looking for a configuration format which had to meet a number of requirements:

• Support a nested hierarchy of elements
• Support a nested hierarchy of elements: the level of nesting of configuration elements should not be arbitrarily restricted. This requirement rules out .ini-like formats, which typically allow no more than two levels of nesting. However, a JSON-like format is a natural fit for this requirement. In this scheme, a configuration consists of key-value mappings, lists and scalar values (strings, numbers, Boolean values and so on). A mapping in JSON is delimited with { and }, and a list is delimited with [ and ]. It’s also sometimes useful to consider a mapping body, which is the part of a mapping between { and }, and a list body, which is the part of a list between [ and ].
• Comments should be allowed
• Comments should be allowed: this is useful to document things in the configuration which aren’t obvious. This requirement rules out standard JSON. Identifiers should be usable as mapping keys: requiring quoted strings, as JSON does, introduces a lot of noise without any real benefit. Of course, strings which aren’t identifiers should be allowed as keys - they just shouldn’t always be required.
• Trailing commas should be allowed
• Trailing commas should be allowed: lists and maps should allow a single trailing comma, as it makes life easier in numerous ways. This requirement also rules out standard JSON.
• Newlines, as well as commas, should be usable to separate elements
• Newlines, as well as commas, should be usable to separate elements: sometimes, newlines are essential in making a configuration more readable. If possible, newlines should be usable as element delimiters, as well as commas which are used in JSON and other similar formats.
• Ordinary strings can be single-quoted
• Ordinary strings can be single-quoted: by ordinary strings, we mean normal strings rather than “raw” strings which don’t support processing of escape sequences such as \n or \t. Such strings should be specifiable using either single- or double-quotes. This allows specification of "O'Flaherty" or '5" floppy' straightforwardly, without the need for escape sequences.
• Full Unicode support
• Full Unicode support: it should be possible to represent any Unicode text in configurations.
• Multi-line strings
• Multi-line strings: It should be possible to specify string literals which span multiple lines in a simple and easy way.
• It should be possible to use hexadecimal numbers
• It should be possible to use hexadecimal numbers: there are situations where hexadecimal numbers (e.g. 0xabcd) confer better readability than decimals. By extension, octal numbers (e.g. 0o377) and binary numbers (e.g. 0b01100111) would also be nice to have.
• Legal JSON should be accepted
• Legal JSON should be accepted: this would ease transition to CFG from JSON configurations, which were/are in common use.
• Allow other configurations to be included
• Allow other configurations to be included: rather than requiring an entire configuration to be in a single file, it would be advantageous to allow configuration files to somehow include other configuration files. This would allow sharing of common component configurations (e.g. logging) across multiple projects, and also allow users to inspect and work on disparate parts of the configuration independently (to some extent).
• Reference one part of a configuration from another
• Reference one part of a configuration from another: it should be possible, in the interests of DRY (Don’t-Repeat-Yourself), to allow a part of a configuration to be referenced from another.
• Allow access to application state
• Allow access to application state: there are situations where it can be useful to pick up configuration values from aspects of the application state, such as environment variables visible to the application, and present them in unified fashion through a single configuration point. This could be extended to other program internals, such as standard process streams (stdin, stdout and stderr, or to allow specification of specific entry points in the program using the configuration).
• Allow composition of parts of a configuration from other parts
• Allow composition of parts of a configuration from other parts: there are often situations where you want to use part of an existing configuration and put it together in some way with other things. Composition should allow operations such as list composition by concatenation and slicing, and mapping composition by merging mappings. Composition might require simple expression evaluation, but without unconstrained evaluation or the need for Turing-completeness.
• Allow a configuration file to specify any of a mapping, a list or a mapping body
• Allow a configuration file to specify any of a mapping, a list or a mapping body: this avoids the need for unnecessary boilerplate, while keeping things unambiguous as far as possible.
• Order independence apart from lists
• Order independence apart from lists: Apart from lists, which are intrinsically ordered, other elements in the configuration should be independent of ordering.
• Copy-and-paste and typo protection
• Copy-and-paste and typo protection: At least some protection should be provided against common copy-and-paste and typographical errors, such as disallowing duplicate keys in a mapping (which can commonly occur due to typos or copy-and-paste-and-forget-to-change-after-pasting).

In 2008, we couldn’t find anything which met these requirements. So we developed this configuration format and a Python module to make use of it. The module was published on the Python Package Index (but not especially strongly publicised). It’s been used on a few internal projects, and had some users on PyPI, but not seen much development after 2010 - because the module met many, though not all, of the above requirements.

Review of other systems

Since 2010, a number of new configuration formats have been proposed: for example, HOCON (2011), JSON5 (2012), TOML (2013), HJSON (2014), and SANE (2018), to name but five.

We investigated these new formats to see which might meet our original requirements. All of them support the requirement to support N-level hierarchical configurations, and to allow the full range of Unicode characters in strings. Our findings are summarised in the table below.

Requirement	JSON	JSON5	HJSON	HOCON	SANE	TOML
Comments allowed?	No	Line, block	Line, block	Line	Line	Line
Allow trailing commas?	No	Yes	Yes	Yes	Yes	Yes
Newlines (as well as commas) can separate elements?	No	No	Yes	Yes	Yes	Yes
Identifiers usable as mapping keys?	No	Yes	Yes	Yes	Yes	Yes
Ordinary strings can be single-quoted?	No	Yes	Yes	No	No	No
Multi-line strings?	No	Yes	Yes	Yes	Yes	Yes
Hexadecimal numbers?	No	Yes	No	No	Yes	Yes
Legal JSON accepted?	Yes	Yes	Yes	Yes	No	No
Include other configurations?	No	No	No	Yes	No	No
Reference parts of a configuration?	No	No	No	Yes	No	No
Allow access to environment variables?	No	No	No	Yes	No	No
Allow composing from different parts of a configuration?	No	No	No	Yes	No	No
Configuration file type	Mapping, List	Mapping	Mapping	Mapping, Mapping body	Mapping body	Mapping body
Order independence?	Yes	Yes	Yes	No	Yes	No

As can be seen from the table above, none of the formats looked at (which are commonly proposed for use in application configuration) meet all the requirements we originally identified. HOCON, which comes closest in terms of number of requirements satisfied or partially satisfied, has numerous perversities which make it seem like it’s not a good fit for our needs: for example, in HOCON the fragment:

3.14 : 42

is interpreted as the key-value pair "3" : { "14" : 42 }, which seems bizarre. Note: TOML has the same behaviour.

Project Status

Following the review, we decided that rather than switching to one of these newer formats, it was worth updating our old implementation to work better with recent Python versions, as well as to provide implementations for other platforms: the JVM (using Kotlin), .NET, Go, Rust, D and JavaScript.

Reporting Problems

If you run into problems, you can raise issues against the issue trackers relating to the relevant implementations using the links below. These are hosted in the relevant source repositories for these implementations.

• Python
• Kotlin/Java
• .NET
• Go
• Rust
• D
• JavaScript