What, another new configuration format?

It’s not a new format. Back in 2008, we were looking for a configuration format which had to meet a number of requirements:

  • Support a nested hierarchy of elements
  • Comments should be allowed
  • Trailing commas should be allowed
  • Newlines, as well as commas, should be usable to separate elements
  • Ordinary strings can be single-quoted
  • Full Unicode support
  • Multi-line strings
  • It should be possible to use hexadecimal numbers
  • Legal JSON should be accepted
  • Allow other configurations to be included
  • Reference one part of a configuration from another
  • Allow access to application state
  • Allow composition of parts of a configuration from other parts
  • Allow a configuration file to specify any of a mapping, a list or a mapping body
  • Order independence apart from lists
  • Copy-and-paste and typo protection

In 2008, we couldn’t find anything which met these requirements. So we developed this configuration format and a Python module to make use of it. The module was published on the Python Package Index (but not especially strongly publicised). It’s been used on a few internal projects, and had some users on PyPI, but not seen much development after 2010 - because the module met many, though not all, of the above requirements.

Review of other systems

Since 2010, a number of new configuration formats have been proposed: for example, HOCON (2011), JSON5 (2012), TOML (2013), HJSON (2014), and SANE (2018), to name but five.

We investigated these new formats to see which might meet our original requirements. All of them support the requirement to support N-level hierarchical configurations, and to allow the full range of Unicode characters in strings. Our findings are summarised in the table below.

Comments allowed? No Line, block Line, block Line Line Line
Allow trailing commas? No Yes Yes Yes Yes Yes
Newlines (as well as commas) can separate elements? No No Yes Yes Yes Yes
Identifiers usable as mapping keys? No Yes Yes Yes Yes Yes
Ordinary strings can be single-quoted? No Yes Yes No No No
Multi-line strings? No Yes Yes Yes Yes Yes
Hexadecimal numbers? No Yes No No Yes Yes
Legal JSON accepted? Yes Yes Yes Yes No No
Include other configurations? No No No Yes No No
Reference parts of a configuration? No No No Yes No No
Allow access to environment variables? No No No Yes No No
Allow composing from different parts of a configuration? No No No Yes No No
Configuration file type Mapping, List Mapping Mapping Mapping, Mapping body Mapping body Mapping body
Order independence? Yes Yes Yes No Yes No

As can be seen from the table above, none of the formats looked at (which are commonly proposed for use in application configuration) meet all the requirements we originally identified. HOCON, which comes closest in terms of number of requirements satisfied or partially satisfied, has numerous perversities which make it seem like it’s not a good fit for our needs: for example, in HOCON the fragment:

3.14 : 42

is interpreted as the key-value pair "3" : { "14" : 42 }, which seems bizarre. Note: TOML has the same behaviour.

Project Status

Following the review, we decided that rather than switching to one of these newer formats, it was worth updating our old implementation to work better with recent Python versions, as well as to provide implementations for other platforms: the JVM (using Kotlin), .NET, Go, Rust, D and JavaScript.

Reporting Problems

If you run into problems, you can raise issues against the issue trackers relating to the relevant implementations using the links below. These are hosted in the relevant source repositories for these implementations.