History¶
What, another new configuration format?¶
It’s not a new format. Back in 2008, we were looking for a configuration format which had to meet a number of requirements:
- Support a nested hierarchy of elements
- Comments should be allowed
- Trailing commas should be allowed
- Newlines, as well as commas, should be usable to separate elements
- Ordinary strings can be single-quoted
- Full Unicode support
- Multi-line strings
- It should be possible to use hexadecimal numbers
- Legal JSON should be accepted
- Allow other configurations to be included
- Reference one part of a configuration from another
- Allow access to application state
- Allow composition of parts of a configuration from other parts
- Allow a configuration file to specify any of a mapping, a list or a mapping body
- Order independence apart from lists
- Copy-and-paste and typo protection
In 2008, we couldn’t find anything which met these requirements. So we developed this configuration format and a Python module to make use of it. The module was published on the Python Package Index (but not especially strongly publicised). It’s been used on a few internal projects, and had some users on PyPI, but not seen much development after 2010 - because the module met many, though not all, of the above requirements.
Review of other systems¶
Since 2010, a number of new configuration formats have been proposed: for example, HOCON (2011), JSON5 (2012), TOML (2013), HJSON (2014), and SANE (2018), to name but five.
We investigated these new formats to see which might meet our original requirements. All of them support the requirement to support N-level hierarchical configurations, and to allow the full range of Unicode characters in strings. Our findings are summarised in the table below.
Requirement | JSON | JSON5 | HJSON | HOCON | SANE | TOML |
---|---|---|---|---|---|---|
Comments allowed? | No | Line, block | Line, block | Line | Line | Line |
Allow trailing commas? | No | Yes | Yes | Yes | Yes | Yes |
Newlines (as well as commas) can separate elements? | No | No | Yes | Yes | Yes | Yes |
Identifiers usable as mapping keys? | No | Yes | Yes | Yes | Yes | Yes |
Ordinary strings can be single-quoted? | No | Yes | Yes | No | No | No |
Multi-line strings? | No | Yes | Yes | Yes | Yes | Yes |
Hexadecimal numbers? | No | Yes | No | No | Yes | Yes |
Legal JSON accepted? | Yes | Yes | Yes | Yes | No | No |
Include other configurations? | No | No | No | Yes | No | No |
Reference parts of a configuration? | No | No | No | Yes | No | No |
Allow access to environment variables? | No | No | No | Yes | No | No |
Allow composing from different parts of a configuration? | No | No | No | Yes | No | No |
Configuration file type | Mapping, List | Mapping | Mapping | Mapping, Mapping body | Mapping body | Mapping body |
Order independence? | Yes | Yes | Yes | No | Yes | No |
As can be seen from the table above, none of the formats looked at (which are commonly proposed for use in application configuration) meet all the requirements we originally identified. HOCON, which comes closest in terms of number of requirements satisfied or partially satisfied, has numerous perversities which make it seem like it’s not a good fit for our needs: for example, in HOCON the fragment:
3.14 : 42
is interpreted as the key-value pair "3" : { "14" : 42 }
, which seems
bizarre. Note: TOML has the same behaviour.
Project Status¶
Following the review, we decided that rather than switching to one of these newer formats, it was worth updating our old implementation to work better with recent Python versions, as well as to provide implementations for other platforms: the JVM (using Kotlin), .NET, Go, Rust, D and JavaScript.
Reporting Problems¶
If you run into problems, you can raise issues against the issue trackers relating to the relevant implementations using the links below. These are hosted in the relevant source repositories for these implementations.