.. meta:: :description: CFG configuration format history :keywords: CFG, configuration, history .. |--| unicode:: U+2013 .. _history: History ======= .. _requirements: What, *another* new configuration format? ----------------------------------------- It's not a new format. Back in `2008 `_, we announced a configuration format which had to meet a number of requirements (click on the little arrows below to see more detail): .. cssclass:: summary-detail * **Support a nested hierarchy of elements**: the level of nesting of configuration elements should not be arbitrarily restricted. This requirement rules out .ini-like formats, which typically allow no more than two levels of nesting. However, a JSON-like format is a natural fit for this requirement. In this scheme, a configuration consists of *key-value mappings*, *lists* and *scalar values* (strings, numbers, Boolean values and so on). A mapping in JSON is delimited with ``{`` and ``}``, and a list is delimited with ``[`` and ``]``. It's also sometimes useful to consider a :term:`mapping body`, which is the part of a mapping between ``{`` and ``}``, and a :term:`list body`, which is the part of a list between ``[`` and ``]``. * **Comments should be allowed**: this is useful to document things in the configuration which aren't obvious. This requirement rules out standard JSON. Identifiers should be usable as mapping keys: requiring quoted strings, as JSON does, introduces a lot of noise without any real benefit. Of course, strings which aren't identifiers *should* be allowed as keys - they just shouldn't always be *required*. * **Trailing commas should be allowed**: lists and maps should allow a single trailing comma, as it makes life easier in numerous ways. This requirement also rules out standard JSON. * **Newlines, as well as commas, should be usable to separate elements**: sometimes, newlines are essential in making a configuration more readable. If possible, newlines should be usable as element delimiters, as well as commas which are used in JSON and other similar formats. * **Ordinary strings can be single-quoted**: by ordinary strings, we mean normal strings rather than “raw” strings which don't support processing of escape sequences such as ``\n`` or ``\t``. Such strings should be specifiable using either single- or double-quotes. This allows specification of ``"O'Flaherty"`` or ``'5" floppy'`` straightforwardly, without the need for escape sequences. * **Full Unicode support**: it should be possible to represent any Unicode text in configurations. * **Multi-line strings**: It should be possible to specify string literals which span multiple lines in a simple and easy way. * **It should be possible to use hexadecimal numbers**: there are situations where hexadecimal numbers (e.g. ``0xabcd``) confer better readability than decimals. By extension, octal numbers (e.g. ``0o377``) and binary numbers (e.g. ``0b01100111``) would also be nice to have. * **Syntactically valid JSON should be accepted**: this would ease transition to CFG from JSON configurations, which were/are in common use. * **Allow other configurations to be included**: rather than requiring an entire configuration to be in a single file, it would be advantageous to allow configuration files to somehow include other configuration files. This would allow sharing of common component configurations (e.g. logging) across multiple projects, and also allow users to inspect and work on disparate parts of the configuration independently (to some extent). * **Reference one part of a configuration from another**: it should be possible, in the interests of DRY (Don't-Repeat-Yourself), to allow a part of a configuration to be referenced from another. * **Allow access to application state**: there are situations where it can be useful to pick up configuration values from aspects of the application state, such as environment variables visible to the application, and present them in unified fashion through a single configuration point. This could be extended to other program internals, such as standard process streams (``stdin``, ``stdout`` and ``stderr``, or to allow specification of specific entry points in the program using the configuration). * **Allow composition of parts of a configuration from other parts**: there are often situations where you want to use part of an existing configuration and put it together in some way with other things. Composition should allow operations such as list composition by concatenation and slicing, and mapping composition by merging mappings. Composition might require simple expression evaluation, but without unconstrained evaluation or the need for Turing-completeness. * **Allow a configuration file to specify any of a mapping, a list or a mapping body**: this avoids the need for unnecessary boilerplate, while keeping things unambiguous as far as possible. * **Order independence apart from lists**: Apart from lists, which are intrinsically ordered, other elements in the configuration should be independent of ordering. * **Copy-and-paste and typo protection**: At least some protection should be provided against common copy-and-paste and typographical errors, such as disallowing duplicate keys in a mapping (which can commonly occur due to typos or copy-and-paste-and-forget-to-change-after-pasting). Back in 2004 when we first started looking, we couldn't find anything which met these requirements. So we developed this configuration format and `a Python module`__ to make use of it. The module was first published_ on the Python Package Index in 2004 (but not especially publicised). It's been used on a few internal projects, and had some users on PyPI, but not seen much development after 2010 - because the module met many, though not all, of the above requirements. Review of other systems ----------------------- Since 2010, a number of new configuration formats have been proposed: for example, HOCON_ (2011), JSON5_ (2012), TOML_ (2013), HJSON_ (2014), and SANE_ (2018, though it now appears to be inactive), to name but five. We investigated these new formats to see which might meet our original requirements. All of them support the requirement to support N-level hierarchical configurations, and to allow the full range of Unicode characters in strings. Our findings are summarised in the table below. .. raw:: html .. csv-table:: :file: reqts.csv :header-rows: 1 :name: reqts-table :class: table table-striped table-bordered :widths: 40, 10, 10, 10, 10, 10, 10, 10 .. _moddoc: https://www.red-dove.com/old_site/config-doc/ __ moddoc_ .. _published: https://pypi.org/project/config/ .. _JSON5: https://json5.org/ .. _HJSON: http://hjson.org/ .. _HOCON: https://github.com/lightbend/config .. _SANE: https://gitlab.com/bloom42/sane .. _TOML: https://github.com/toml-lang/toml As can be seen from the table above, none of the other formats looked at (which are commonly proposed for use in application configuration) meet all the requirements we originally identified. HOCON, which comes closest in terms of number of requirements satisfied or partially satisfied, has numerous perversities which make it seem like it's not a good fit for our needs: for example, in HOCON the fragment:: 3.14 : 42 is interpreted as the key-value pair ``"3" : { "14" : 42 }``, which seems bizarre. Note: TOML has the same behaviour. We didn't look at configuration systems that are like programming languages, such as `Dhall `_, `CUE `_ or `Jsonnet `_ |--| they are a different kind of thing. Project Status -------------- Following the review, we decided that rather than switching to one of these newer formats, it was worth :ref:`updating our old implementation` to work better with recent Python versions, as well as to provide implementations for other platforms: :ref:`the JVM ` (using Kotlin), :ref:`.NET `, :ref:`Go `, :ref:`Rust `, :ref:`D `, :ref:`JavaScript `, :ref:`Ruby `, :ref:`Elixir `, :ref:`Nim ` and :ref:`Dart `. As well as providing useful functionality for application and library configuration in a range of environments, these implementations also serve as an example of comparative development in these various languages of a set of functionality which is non-trivial, but not so large as to be difficult to assimilate / understand. Reporting Problems ------------------ If you run into problems, you can raise issues against the issue trackers relating to the relevant implementations using the links below. These are hosted in the relevant source repositories for these implementations. * `Report a problem with the Python implementation `_ * `Report a problem with the Kotlin/Java implementation `_ * `Report a problem with the .NET implementation `_ * `Report a problem with the Go implementation `_ * `Report a problem with the Rust implementation `_ * `Report a problem with the D implementation `_ * `Report a problem with the JavaScript implementation `_ * `Report a problem with the Ruby implementation `_ * `Report a problem with the Elixir implementation `_ * `Report a problem with the Nim implementation `_ * `Report a problem with the Dart implementation `_