python-anyconfig

Usage

Here are some code examples of API usage.

Loading single config file

Here are some example code to load single config file:

import anyconfig

# Config type (format) is automatically detected by filename (file
# extension).
data1 = anyconfig.load("/path/to/foo/conf.d/a.yml")

# Loaded config data is a dict-like object.
# examples:
# data1["a"] => 1
# data1["b"]["b1"] => "xyz"
# data1["c"]["c1"]["c13"] => [1, 2, 3]

# Same as above but I recommend to use the former.
data2 = anyconfig.single_load("/path/to/foo/conf.d/a.yml")

# Or you can specify config type explicitly as needed.
cnf_path = "/path/to/foo/conf.d/b.conf"
data3 = anyconfig.load(cnf_path, ac_parser="yaml")

# Same as above but ...
data4 = anyconfig.single_load(cnf_path, ac_parser="yaml")

# Same as above as a result but make parser instance and pass it explicitly.
yml_psr = anyconfig.find_loader(None, ac_parser="yaml")
data5 = anyconfig.single_load(cnf_path, yml_psr)  # Or: anyconfig.load(...)

# Same as above as a result but make parser instance and pass it explicitly.
yml_psr = anyconfig.find_loader(None, ac_parser="yaml")
data6 = anyconfig.single_load(cnf_path, yml_psr)  # Or: anyconfig.load(...)

# Similar to the previous examples but parser is specified explicitly to use
# ruamel.yaml based YAML parser instead of PyYAML based one, and give
# ruamel.yaml specific option.
data7 = anyconfig.load(cnf_path, ac_parser="ruamel.yaml",
                       allow_duplicate_keys=True)

# Same as above but open the config file explicitly before load.
with anyconfig.open("/path/to/foo/conf.d/a.yml") as istrm:
    data10 = anyconfig.load(istrm)

# Same as above but with specifying config type explicitly.
with anyconfig.open("/path/to/foo/conf.d/a.yml", ac_parser="yaml") as istrm:
    data11 = anyconfig.load(istrm)

Exceptions raised on load

Exception may be raised if something goes wrong. Then, you have to catch them if you want to process more w/ errors ignored or handled.

>>> import anyconfig
>>> anyconfig.single_load(None)
Traceback (most recent call last):
  ...
ValueError: path_or_stream or forced_type must be some value
>>> anyconfig.single_load(None, ac_parser="backend_module_not_avail")
Traceback (most recent call last):
  ...
anyconfig.backends.UnknownParserTypeError: No parser found for type 'backend_module_not_avail'
>>> anyconfig.single_load(None, ac_parser="not_existing_type")
Traceback (most recent call last):
  ...
anyconfig.backends.UnknownParserTypeError: No parser found for type 'not_existing_type'
>>> anyconfig.single_load("unknown_type_file.conf")
Traceback (most recent call last):
  ...
anyconfig.backends.UnknownFileTypeError: No parser found for file 'unknown_type_file.conf'
>>>

Common and backend specific Keyword options on load single config file

Here is a brief summary of keyword options prefixed with ‘ac_’ to change the behavior on load.

Option	Type	Note
ac_parser	str or anyconfig.backend.base.Parser	Forced parser type or parser object
ac_dict	callable	Any callable (function or class) to make mapping object will be returned as a result or None. If not given or ac_dict is None, default mapping object used to store resutls is dict or OrderedDict if ac_ordered is True and selected backend can keep the order of items in mapping objects.
ac_ordered	bool	True to keep resuls ordered. Please note that order of items in results may be lost depends on backend used.
ac_template	bool	Assume given file may be a template file and try to compile it AAR if True
ac_context	mapping object	Mapping object presents context to instantiate template
ac_schema	str	JSON schema file path to validate given config file
ac_query	str	JMESPath expression to query data

You can pass backend (config loader) specific keyword options to these load and dump functions as needed along with the above anyconfig specific keyword options:

# from python -c "import json; help(json.load)":
# Help on function load in module json:
#
# load(fp, encoding=None, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw)
#    Deserialize ``fp`` (a ``.read()``-supporting file-like object containing
#    a JSON document) to a Python object.
#    ...
data6 = anyconfig.load("foo.json", parse_float=None)

Allowed keyword options depend on backend, so please take a look at each backend API docs for more details about it.

Others topics on load

Anyconfig also enables:

• to load a config which is actually a Jinja2 1 template file, the file will be rendered before load. See Template config support section for more details.

• to validate a config file with a JSON schema 2 before load. See Validation with and/or generate JSON Schema section for more details.

• to search and filter results with a JMESPath expression 3 after load. See Query results with JMESPath expression section for more details.

Note

The returned object is a mapping object, dict or collections.OrderedDict object by default.

1

http://jinja.pocoo.org

2

http://json-schema.org

3

http://jmespath.org

Loading multiple config files

Here are some example code to load multiple config files:

import anyconfig

# Specify config files by list of paths:
data1 = anyconfig.load(["/etc/foo.d/a.json", "/etc/foo.d/b.json"])

# Similar to the above but all or one of config files are missing:
data2 = anyconfig.load(["/etc/foo.d/a.json", "/etc/foo.d/b.json"],
                       ignore_missing=True)

# Specify config files by glob path pattern:
cnf_path = "/etc/foo.d/*.json"
data3 = anyconfig.load(cnf_path)

# Similar to above but make parser instance and pass it explicitly.
psr = anyconfig.find_loader(cnf_path)
data4 = anyconfig.load(cnf_path, psr)

# Similar to the above but parameters in the former config file will be simply
# overwritten by the later ones:
data5 = anyconfig.load("/etc/foo.d/*.json", ac_merge=anyconfig.MS_REPLACE)

Strategies to merge data loaded from multiple config files

On loading multiple config files, you can choose ‘strategy’ to merge configurations from the followings and pass it with ac_merge keyword option:

• anyconfig.MS_REPLACE: Replace all configuration parameter values provided in former config files are simply replaced w/ the ones in later config files.

  For example, if a.yml and b.yml are like followings:

  a.yml:

  a: 1
  b:
     - c: 0
     - c: 2
  d:
     e: "aaa"
     f: 3
  

  b.yml:

  b:
     - c: 3
  d:
     e: "bbb"
  

  then:

  load(["a.yml", "b.yml"], ac_merge=anyconfig.MS_REPLACE)
  

  will give object such like:

  {'a': 1, 'b': [{'c': 3}], 'd': {'e': "bbb"}}
  

• anyconfig.MS_NO_REPLACE: Do not replace configuration parameter values provided in former config files.

  For example, if a.yml and b.yml are like followings:

  a.yml:

  b:
     - c: 0
     - c: 2
  d:
     e: "aaa"
     f: 3
  

  b.yml:

  a: 1
  b:
     - c: 3
  d:
     e: "bbb"
  

  then:

  load(["a.yml", "b.yml"], ac_merge=anyconfig.MS_NO_REPLACE)
  

  will give object such like:

  {'a': 1, 'b': [{'c': 0}, {'c': 2}], 'd': {'e': "bbb", 'f': 3}}
  

• anyconfig.MS_DICTS (default): Merge dicts recursively. That is, the following:

  load(["a.yml", "b.yml"], ac_merge=anyconfig.MS_DICTS)
  

  will give object such like:

  {'a': 1, 'b': [{'c': 3}], 'd': {'e': "bbb", 'f': 3}}
  

  This is the merge strategy chosen by default.

• anyconfig.MS_DICTS_AND_LISTS: Merge dicts and lists recursively. That is, the following:

  load(["a.yml", "b.yml"], ac_merge=anyconfig.MS_DICTS_AND_LISTS)
  

  will give object such like:

  {'a': 1, 'b': [{'c': 0}, {'c': 2}, {'c': 3}], 'd': {'e': "bbb", 'f': 3}}
  

Or you you can implement custom function or class or anything callables to merge nested dicts by yourself and utilize it with ac_merge keyword option like this:

def my_merge_fn(self, other, key, val=None, **options):
    """
    :param self: mapping object to update with `other`
    :param other: mapping object to update `self`
    :param key: key of mapping object to update
    :param val: value to update self alternatively

    :return: None but `self` will be updated
    """
    if key not in self:
        self[key] = other[key] if val is None else val

load(["a.yml", "b.yml"], ac_merge=my_merge_fn)

Please refer to the existing functions in anyconfig.dicsts (_update_* functions) to implement custom functions to merge nested dicts for more details.

Common and backend specific Keyword options on load multiple config files

Here is a brief summary of keyword options prefixed with ‘ac_’ in addition to the keyword options explained in the Common and backend specific Keyword options on load single config file section to change the behavior on load multiple files.

Option	Type	Note
ac_merge	str	One of anyconfig.dicts.MERGE_STRATEGIES to select strategy of how to merge results loaded from multiple configuration files. See the doc of anyconfig.dicts for more details of strategies. The default is anyconfig.dicts.MS_DICTS.
ac_marker	str	Glob marker string to detect paths patterns. ‘*’ by default.

Dumping config data

A pair of APIs are provided to dump config data loaded w/ using loading APIs as described previously and corresponding to them.

• dumps(): Dump data as a string

• dump(): Dump data to file of which path was given or file-like object opened

Note

To specify the format or backend type w/ ac_parser keyword option is necessary for dumps() API because anyconfig cannot determine the type w/o it.

Like loading APIs, you can pass common and backend specific keyword options to them.

• common keyword options: ac_parser to determine which backend to use

• backend specific keyword options: see each backends’ details

Here are some examples of these usage:

In [1]: s = """a: A
   .....: b:
   .....:   - b0: 0
   .....:   - b1: 1
   .....: c:
   .....:   d:
   .....:     e: E
   .....: """

In [2]: cnf = anyconfig.loads(s, ac_parser="yaml")

In [3]: cnf
Out[3]: {'a': 'A', 'b': [{'b0': 0}, {'b1': 1}], 'c': {'d': {'e': 'E'}}}

In [4]: anyconfig.dumps(cnf, ac_parser="yaml")  # ac_parser option is necessary.
Out[4]: 'a: A\nc:\n  d: {e: E}\nb:\n- {b0: 0}\n- {b1: 1}\n'

In [5]: print(anyconfig.dumps(cnf, ac_parser="yaml"))
a: A
c:
  d: {e: E}
b:
- {b0: 0}
- {b1: 1}

In [6]: print(anyconfig.dumps(cnf, ac_parser="json"))
{"a": "A", "c": {"d": {"e": "E"}}, "b": [{"b0": 0}, {"b1": 1}]}

In [7]: print(anyconfig.dumps(cnf, ac_parser="ini"))  # It cannot!
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-228-2b2771a44a7e> in <module>()
----> 1 print(anyconfig.dumps(cnf, ac_parser="ini"))
    ...
AttributeError: 'str' object has no attribute 'iteritems'

In [8]: print(anyconfig.dumps(cnf, ac_parser="configobj"))
a = A
b = {'b0': 0}, {'b1': 1}
[c]
[[d]]
e = E

In [9]:

Like this example, it’s not always possible to dump data to any formats because of limitations of formarts and/or backends.

Keep the order of configuration items

If you want to keep the order of configuration items, specify ac_order=True on load or specify ac_dict to any mapping object can save the order of items such like collections.OrderedDict (or OrderedDict). Otherwise, the order of configuration items will be lost by default.

Please note that anyconfig.load APIs sometimes cannot keep the order of items in the original data even if ac_order=True or ac_dict=<ordereddict> was specified because used backend or module cannot keep that. For example, JSON backend can keep items but current YAML backend does not due to the limitation of YAML module it using.

Validation with and/or generate JSON Schema

If jsonschema 4 is installed and available, you can validate config files with using anyconfig.validate() since 0.0.10.

# Validate a JSON config file (conf.json) with JSON schema (schema.json).
# If validatation succeeds, `rc` -> True, `err` -> ''.
conf1 = anyconfig.load("/path/to/conf.json")
schema1 = anyconfig.load("/path/to/schema.json")
(rc, err) = anyconfig.validate(conf1, schema1)

# Similar to the above but both config and schema files are in YAML.
conf2 = anyconfig.load("/path/to/conf.yml")
schema2 = anyconfig.load("/path/to/schema.yml")
(rc, err) = anyconfig.validate(conf2, schema2)

# Similar to the above but exception will be raised if validation fails.
(rc, _err) = anyconfig.validate(conf2, schema2, ac_schema_safe=False)

It’s also able to validate config files during load:

# Validate a config file (conf.yml) with JSON schema (schema.yml) while
# loading the config file.
conf1 = anyconfig.load("/a/b/c/conf.yml", ac_schema="/c/d/e/schema.yml")

# Validate config loaded from multiple config files with JSON schema
# (schema.json) while loading them.
conf2 = anyconfig.load("conf.d/*.yml", ac_schema="/c/d/e/schema.json")

And even if you don’t have any JSON schema files, don’t worry ;-), anyconfig can generate the schema for your config files on demand and you can save it in any formats anyconfig supports.

# Generate a simple JSON schema file from config file loaded.
conf1 = anyconfig.load("/path/to/conf1.json")
schema1 = anyconfig.gen_schema(conf1)
anyconfig.dump(schema1, "/path/to/schema1.yml")

# Generate more strict (precise) JSON schema file from config file loaded.
schema2 = anyconfig.gen_schema(conf1, ac_schema_strict=True)
anyconfig.dump(schema2, "/path/to/schema2.json")

Note

If you just want to generate JSON schema from your config files, then you don’t need to install jsonschema in advance because anyconfig can generate JSON schema without jsonschema module.

4

https://pypi.python.org/pypi/jsonschema

Template config support

anyconfig supports template config files since 0.0.6. That is, config files written in Jinja2 template 5 will be compiled before loading w/ backend module.

Note

Template config support is disabled by default to avoid side effects when processing config files of jinja2 template or having some expressions similar to jinaj2 template syntax.

Anyway, a picture is worth a thousand words. Here is an example of template config files.

ssato@localhost% cat a.yml
a: 1
b:
  {% for i in [1, 2, 3] -%}
  - index: {{ i }}
  {% endfor %}
{% include "b.yml" %}
ssato@localhost% cat b.yml
c:
  d: "efg"
ssato@localhost% anyconfig_cli a.yml --template -O yaml -s
a: 1
b:
- {index: 1}
- {index: 2}
- {index: 3}
c: {d: efg}
ssato@localhost%

And another one:

In [1]: import anyconfig

In [2]: ls *.yml
a.yml  b.yml

In [3]: cat a.yml
a: {{ a }}
b:
  {% for i in b -%}
  - index: {{ i }}
  {% endfor %}
{% include "b.yml" %}

In [4]: cat b.yml
c:
  d: "efg"

In [5]: context = dict(a=1, b=[2, 4])

In [6]: anyconfig.load("*.yml", ac_template=True, ac_context=context)
Out[6]: {'a': 1, 'b': [{'index': 2}, {'index': 4}], 'c': {'d': 'efg'}}

5

Jinja2 template engine (http://jinja.pocoo.org) and its language (http://jinja.pocoo.org/docs/dev/)

Query results with JMESPath expression

anyconfig supports to query result mapping object with JMESPath expression since 0.8.3 like the following example 6 .

>>> yaml_s = """\
... locations:
...   - name: Seattle
...     state: WA
...   - name: New York
...     state: NY
...   - name: Olympia
...     state: WA
... """
>>> query = "locations[?state == 'WA'].name | sort(@) | {WashingtonCities: join(', ', @)}"
>>> anyconfig.loads(yaml_s, ac_parser="yaml", ac_query=query)
 {'WashingtonCities': 'Olympia, Seattle'}
>>>

Different from other libraries can process JMESPath expressions, anyconfig can query data of any formats it supports, with help of the jmespath support library 7 . That is, you can query XML, YAML, BSON, Toml, and, of course JSON files with JMESPath expression.

6

This example is borrowed from JMESPath home, http://jmespath.org

7

https://github.com/jmespath/jmespath.py

Other random topics with API usage

Suppress logging messages from anyconfig module

anyconfig uses a global logger named anyconfig and logging messages are suppressed by default as NullHandler was attached to the logger 8 . If you want to see its log messages out, you have to configure it (add handler and optionally set log level) like the followings.

• Set the log level and handler of anyconfig module before load to print log messages such as some backend modules are not available, when it’s initialized:

In [1]: import logging

In [2]: LOGGER = logging.getLogger("anyconfig")

In [3]: LOGGER.addHandler(logging.StreamHandler())

In [4]: LOGGER.setLevel(logging.ERROR)

In [5]: import anyconfig

In [6]: anyconfig.dumps(dict(a=1, b=[1,2]), "aaa")
No parser found for given type: aaa
Out[6]: '{"a": 1, "b": [1, 2]}'

In [7]:

• Set log level of anyconfig module after load:

In [1]: import anyconfig, logging

In [2]: LOGGER = logging.getLogger("anyconfig")

In [3]: LOGGER.addHandler(logging.StreamHandler())

In [4]: anyconfig.dumps(dict(a=2, b=[1,2]), "unknown_type")
No parser found for given type: unknown_type
Parser unknown_type was not found!
Dump method not implemented. Fallback to json.Parser
Out[4]: '{"a": 2, "b": [1, 2]}'

In [5]:

8

https://docs.python.org/2/howto/logging.html#library-config

Combination with other modules

anyconfig can be combined with other modules such as pyxdg and appdirs 9 .

For example, you can utilize anyconfig and pyxdg or appdirs in you application software to load user config files like this:

import anyconfig
import appdirs
import os.path
import xdg.BaseDirectory

APP_NAME = "foo"
APP_CONF_PATTERN = "*.yml"


def config_path_by_xdg(app=APP_NAME, pattern=APP_CONF_PATTERN):
    return os.path.join(xdg.BaseDirectory.save_config_path(app), pattern)


def config_path_by_appdirs(app=APP_NAME, pattern=APP_CONF_PATTERN):
    os.path.join(appdirs.user_config_dir(app), pattern)


def load_config(fun=config_path_by_xdg):
    return anyconfig.load(fun())

9

http://freedesktop.org/wiki/Software/pyxdg/

10

https://pypi.python.org/pypi/appdirs/

Default config values

Current implementation of anyconfig.*load*() do not provide a way to provide some sane default configuration values (as a dict parameter for example) before/while loading config files. Instead, you can accomplish that by a few lines of code like the followings:

import anyconfig

conf = dict(foo=0, bar='1', baz=[2, 3])  # Default values
conf_from_files = anyconfig.load("/path/to/config_files_dir/*.yml")
anyconfig.merge(conf, conf_from_files)  # conf will be updated.

# Use `conf` ...

or:

conf = dict(foo=0, bar='1', baz=[2, 3])
anyconfig.merge(conf, anyconfig.load("/path/to/config_files_dir/*.yml"))

Environment Variables

It’s a piece of cake to use environment variables as config default values like this:

conf = os.environ.copy()
anyconfig.merge(conf, anyconfig.load("/path/to/config_files_dir/*.yml"))

Load from compressed files

Since 0.2.0, python-anyconfig can load configuration from file or file-like object, called stream internally. And this should help loading configurations from compressed files.

• Loading from a compressed JSON config file:

import gzip

strm = gzip.open("/path/to/gzip/compressed/cnf.json.gz")
cnf = anyconfig.load(strm, "json")

• Loading from some compressed JSON config files:

import gzip
import glob

cnfs = "/path/to/gzip/conf/files/*.yml.gz"
strms = [gzip.open(f) for f in sorted(glob.glob(cnfs))]
cnf = anyconfig.load(strms, "yaml")

Please note that “json” argument passed to anyconfig.load is necessary to help anyconfig find out the configuration type of the file.

Convert from/to bunch objects

It’s easy to convert result conf object from/to bunch objects 10 as anyconfig.load{s,} return a dict-like object:

import anyconfig
import bunch

conf = anyconfig.load("/path/to/some/config/files/*.yml")
bconf = bunch.bunchify(conf)
bconf.akey = ...  # Overwrite a config parameter.
   ...
anyconfig.dump(bconf.toDict(), "/tmp/all.yml")

11

bunch: https://pypi.python.org/pypi/bunch/

Code Reference

anyconfig.api

anyconfig.api.datatypes

anyconfig.api._dump

anyconfig.api._load

anyconfig.api._open

anyconfig.api.utils

anyconfig.backend

anyconfig.backend.base

anyconfig.backend.base.compat

anyconfig.backend.base.datatypes

anyconfig.backend.base.dumpers

anyconfig.backend.base.loaders

anyconfig.backend.base.parsers

anyconfig.backend.base.utils

anyconfig.backend.ini

anyconfig.backend.json

anyconfig.backend.json.common

anyconfig.backend.json.default

anyconfig.backend.json.simplejson

anyconfig.backend.pickle

anyconfig.backend.properties

anyconfig.backend.shellvars

anyconfig.backend.toml

anyconfig.backend.yaml

anyconfig.backend.yaml.common

anyconfig.backend.yaml.pyyaml

anyconfig.backend.yaml.ruamel_yaml

anyconfig.backend.xml

anyconfig.cli

anyconfig.dicts

anyconfig.common

anyconfig.common.datatypes

anyconfig.common.errors

anyconfig.ioinfo

anyconfig.ioinfo.constants

anyconfig.ioinfo.datatypes

anyconfig.ioinfo.detectors

anyconfig.ioinfo.factory

anyconfig.ioinfo.utils

anyconfig.models

anyconfig.models.processor

anyconfig.parser

anyconfig.parsers

anyconfig.parsers.parsers

anyconfig.parsers.utils

anyconfig.processors

anyconfig.processors.datatypes

anyconfig.processors.processors

anyconfig.processors.utils

anyconfig.query

anyconfig.query.datatypes

anyconfig.query.default

anyconfig.query.query

anyconfig.schema

anyconfig.schema.datatypes

anyconfig.schema.default

anyconfig.schema.jsonschema

anyconfig.template

anyconfig.template.jinja2

anyconfig.utils

anyconfig.utils.detectors

anyconfig.utils.files

anyconfig.utils.lists

anyconfig.utils.utils

anyconfig

Philosophy and design principles of anyconfig

Philosophy behind anyconfig

The reason I made anyconfig is aimed to eliminate the need of manual edit of configuration files, which application developers provide in the applications originally, by uesrs. It enables that application developers provide the default configuration and allow users to customize configuration without direct modification of these configuration files at the same time 1 .

There are still many applications force users to edit configuration files directly if any customization was needed. Sometimes application provides special configuration tool to hide this fact from users but most tools try to modify configuration files directly in the end 2 .

With using anyconfig or similar library can merge configuration files, users of applications can customize the behavior of applications by just creation of new configuration files to override the default and there is no need to modify the default ones.

1

One of examples accomplishing this is systemd; systemd allows users to customize the default provided by unit definitions under /usr/lib/systemd/ by putting unit files under /etc/systemd/. Other examples are sysctl (/etc/sysctl.d/) and sudo (/etc/sudoers.d/).

2

I saw openstack provides such tool, openstack-config; IMHO, the problem should be resolved not by such tools but application design itself, that is, openstack should provide the way to override the default by some configuration files users created newly.

Design principle of anyconfig

I try to make anyconfig as thin as possible, that is , it works as a thin wrapper for the backends actually does configuration load and dump. Thus, anyconfig does not try to catch any exceptions such as IOError (ex. it failed to open configuration files) and OSError (ex. you’re not allowed to open configuration files) during load and dump of configuration files. You have to process these exceptions as needed.

And sometimes backend has specific options for load and dump functions, therefore, I make anyconfig to pass these options filtered but un-touched. Filters are statically defined in each backend implementation and maybe lack of some options. Please feel free to report if you find such backend specific options not supported in anyconfig’s corresponding backend implementation and I’ll make them supported.

CLI Usage

python-anyconfig contains a CLI frontend ‘anyconfig_cli’ to demonstrate the power of this library.

It can process config files in any formats supported in your environment and:

• output merged/converted config outputs w/ modifications needed

• output schema file for given inputs

• merge/convert input config and extract part of the config

ssato@localhost% anyconfig_cli -h
Usage: anyconfig_cli [Options...] CONF_PATH_OR_PATTERN_0 [CONF_PATH_OR_PATTERN_1 ..]

Examples:
  anyconfig_cli --list  # -> Supported config types: configobj, ini, json, ...
  # Merge and/or convert input config to output config [file]
  anyconfig_cli -I yaml -O yaml /etc/xyz/conf.d/a.conf
  anyconfig_cli -I yaml '/etc/xyz/conf.d/*.conf' -o xyz.conf --otype json
  anyconfig_cli '/etc/xyz/conf.d/*.json' -o xyz.yml \
    --atype json -A '{"obsoletes": "sysdata", "conflicts": "sysdata-old"}'
  anyconfig_cli '/etc/xyz/conf.d/*.json' -o xyz.yml \
    -A obsoletes:sysdata;conflicts:sysdata-old
  anyconfig_cli /etc/foo.json /etc/foo/conf.d/x.json /etc/foo/conf.d/y.json
  anyconfig_cli '/etc/foo.d/*.json' -M noreplace
  # Get/set part of input config
  anyconfig_cli '/etc/foo.d/*.json' --get a.b.c
  anyconfig_cli '/etc/foo.d/*.json' --set a.b.c=1

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -o OUTPUT, --output=OUTPUT
                        Output file path
  -I ITYPE, --itype=ITYPE
                        Select type of Input config files from configobj, ini,
                        json, msgpack, xml, yaml [Automatically detected by
                        file ext]
  -O OTYPE, --otype=OTYPE
                        Select type of Output config files from configobj,
                        ini, json, msgpack, xml, yaml [Automatically detected
                        by file ext]
  -M MERGE, --merge=MERGE
                        Select strategy to merge multiple configs from
                        replace, noreplace, merge_dicts, merge_dicts_and_lists
                        [merge_dicts]
  -A ARGS, --args=ARGS  Argument configs to override
  --atype=ATYPE         Explicitly select type of argument to provide configs
                        from configobj, ini, json, msgpack, xml, yaml.  If
                        this option is not set, original parser is used: 'K:V'
                        will become {K: V}, 'K:V_0,V_1,..' will become {K:
                        [V_0, V_1, ...]}, and 'K_0:V_0;K_1:V_1' will become
                        {K_0: V_0, K_1: V_1} (where the tyep of K is str, type
                        of V is one of Int, str, etc.
  -x, --ignore-missing  Ignore missing input files
  -T, --template        Enable template config support
  -E, --env             Load configuration defaults from environment values
  -S SCHEMA, --schema=SCHEMA
                        Specify Schema file[s] path
  -s, --silent          Silent or quiet mode
  -q, --quiet           Same as --silent option
  -v, --verbose         Verbose mode

  List specific options:
    -L, --list          List supported config types

  Schema specific options:
    --validate          Only validate input files and do not output. You must
                        specify schema file with -S/--schema option.
    --gen-schema        Generate JSON schema for givne config file[s] and
                        output it instead of (merged) configuration.

  Get/set options:
    -Q QUERY, --query=QUERY
                        Query with JMESPath expression language. See
                        http://jmespath.org for more about JMESPath
                        expression. This option is not used with --get option
                        at the same time. Please note that python module to
                        support JMESPath expression
                        (https://pypi.python.org/pypi/jmespath/) is required
                        to use this option
    --get=GET           Specify key path to get part of config, for example, '
                        --get a.b.c' to config {'a': {'b': {'c': 0, 'd': 1}}}
                        gives 0 and '--get a.b' to the same config gives {'c':
                        0, 'd': 1}.
    --set=SET           Specify key path to set (update) part of config, for
                        example, '--set a.b.c=1' to a config {'a': {'b': {'c':
                        0, 'd': 1}}} gives {'a': {'b': {'c': 1, 'd': 1}}}.
ssato@localhost%

List supported config types (formats)

anyconfig_cli lists config types (formats) supported in your environment with -L/–list option:

$ anyconfig_cli -L
Supported config types: configobj, ini, json, msgpack, xml, yaml
$ anyconfig_cli --list
Supported config types: configobj, ini, json, msgpack, xml, yaml
$

Merge and/or convert input config

anyconfig_cli can process a config file or config files and output merged config in various formats it can support in your environment.

Here are some such examples.

• single input config file, input type is automatically detected from the input file’s extension:

$ cat /tmp/a.yml
a: 1
b:
  c:
    - aaa
    - bbb
d:
  e:
    f: xyz
    g: true
$ anyconfig_cli -O json /tmp/a.yml
Loading: /tmp/a.yml
{"a": 1, "b": {"c": ["aaa", "bbb"]}, "d": {"e": {"g": true, "f": "xyz"}}}

• single input config file with the input type and output option:

$ diff -u /tmp/a.{yml,conf}
$ anyconfig_cli -I yaml -O configobj /tmp/a.conf -o /tmp/a.ini --silent
$ cat /tmp/a.ini
a = 1
[b]
c = aaa, bbb
[d]
[[e]]
g = True
f = xyz
$

• multiple input config files:

$ cat /tmp/b.yml
b:
  i:
    j: 123
d:
  e:
    g: hello, world
l: -1
$ anyconfig_cli /tmp/{a,b}.yml --silent
a: 1
b:
  c: [aaa, bbb]
  i: {j: 123}
d:
  e: {f: xyz, g: 'hello, world'}
l: -1

$

• multiple input config files with merge strategy option:

$ anyconfig_cli /tmp/{a,b}.yml -M replace --silent
a: 1
b:
  i: {j: 123}
d:
  e: {g: 'hello, world'}
l: -1

$

• multiple input config files with template option:

$ cat /tmp/c.yml
m: {{ d.e.g }}
n: {{ b.i.j }}
$ anyconfig_cli /tmp/{a,b,c}.yml --silent --template
a: 1
b:
  c: [aaa, bbb]
  i: {j: 123}
d:
  e: {f: xyz, g: 'hello, world'}
l: -1
m: hello, world
n: 123

$ ls /tmp/*.yml
/tmp/a.yml  /tmp/b.yml  /tmp/c.yml
$ # Same as the privious one but inputs are given in a glob pattern.
$ anyconfig_cli '/tmp/*.yml' --silent --template  # same as the privious one
a: 1
b:
  c: [aaa, bbb]
  i: {j: 123}
d:
  e: {f: xyz, g: 'hello, world'}
l: -1
m: hello, world
n: 123

$

• Missing input config files:

$ ls /tmp/not-exist-file.yml
ls: cannot access /tmp/not-exist-file.yml: No such file or directory
$ anyconfig_cli --ignore-missing /tmp/not-exist-file.yml -s
{}

$ anyconfig_cli --ignore-missing /tmp/not-exist-file.yml -s -A "a: aaa"
No config type was given. Try to parse...
{a: aaa}

$ anyconfig_cli --ignore-missing /tmp/not-exist-file.yml -s -A "a: aaa; b: 123"
No config type was given. Try to parse...
{a: aaa, b: 123}

$

Schema generation and validation

anyconfig_cli can process input config file[s] and generate JSON schema file to validate the config like this:

• An usage example of schema generation option –gen-schema of anyconfig_cli:

$ cat /tmp/a.yml
a: 1
b:
  c:
    - aaa
    - bbb
d:
  e:
    f: xyz
    g: true
$ anyconfig_cli --gen-schema /tmp/a.yml -s -o /tmp/a.schema.json
$ jq '.' /tmp/a.schema.json
{
  "properties": {
    "d": {
      "properties": {
        "e": {
          "properties": {
            "f": {
              "type": "string"
            },
            "g": {
              "type": "boolean"
            }
          },
          "type": "object"
        }
      },
      "type": "object"
    },
    "b": {
      "properties": {
        "c": {
          "type": "array",
          "items": {
            "type": "string"
          }
        }
      },
      "type": "object"
    },
    "a": {
      "type": "integer"
    }
  },
  "type": "object"
}
$

• and schema validation option –validate (and –schema) of anyconfig_cli:

$ anyconfig_cli -A 'a: aaa' --atype yaml /tmp/a.yml -o /tmp/a2.yml --silent
$ head -n 1 /tmp/a.yml
a: 1
$ head -n 1 /tmp/a2.yml
a: aaa
$ anyconfig_cli --validate --schema /tmp/a.schema.json /tmp/a.yml
Loading: /tmp/a.schema.json
Loading: /tmp/a.yml
Validation succeeds
$ anyconfig_cli --validate --schema /tmp/a.schema.json /tmp/a.yml -s; echo $?
0
$ anyconfig_cli --validate --schema /tmp/a.schema.json /tmp/a2.yml -s; echo $?
'aaa' is not of type u'integer'

Failed validating u'type' in schema[u'properties'][u'a']:
    {u'type': u'integer'}

On instance[u'a']:
    'aaa'
Validation failed1
$

Query/Get/set - extract or set part of input config

Here is usage examples of –get option of anyconfig_cli:

$ cat /tmp/a.yml
a: 1
b:
  c:
    - aaa
    - bbb
d:
  e:
    f: xyz
    g: true
$ anyconfig_cli /tmp/a.yml --get d.e.f --silent
xyz
$ anyconfig_cli /tmp/a.yml --get b.c --silent
['aaa', 'bbb']
$ anyconfig_cli /tmp/a.yml --query d.e.g --silent
True
$ anyconfig_cli /tmp/a.yml --query 'b.c[::-1]' --silent
['bbb', 'aaa']

and an usage example of –set option of anyconfig_cli with same input:

$ anyconfig_cli /tmp/a.yml --set "d.e.g=1000" --set "b.c=ccc," --silent
a: 1
b:
  c: [ccc]
d:
  e: {f: xyz, g: true}

$

Introduction

python-anyconfig 1 is a python library provides common APIs to load and dump configuration files in various formats with some useful features such as contents merge, templates, query, schema validation and generation support.

• Home: https://github.com/ssato/python-anyconfig

• Author: Satoru SATOH <satoru.satoh@gmail.com>

• License: MIT licensed

• Document: http://python-anyconfig.readthedocs.org/en/latest/

• Download:

  • PyPI: https://pypi.python.org/pypi/anyconfig

  • Copr RPM repos: https://copr.fedoraproject.org/coprs/ssato/python-anyconfig/

1

This name took an example from the ‘anydbm’ python standard library.

Features

python-anyconfig provides very simple and unified APIs to process configuration files in various formats and related functions:

• Loading configuration files:

  anyconfig.load (path_specs, ac_parser=None, ac_dict=None, ac_template=False, ac_context=None, **options)

  loads configuration data from path_specs. path_specs may be a list of file paths, files or file-like objects, ~pathlib.Path class object, a namedtuple ~anyconfig.globals.IOInfo objects represents some inputs to load data from, and return a dict or dict like object, or a primitive types’ data other than dict represents loaded configuration.

  anyconfig.loads (content, ac_parser=None, ac_dict=None, ac_template=False, ac_context=None, **options)

  loads configuration data from a string just like json.loads does.

• Dumping configuration files:

  anyconfig.dump (data, out, ac_parser=None, **options)

  dumps a configuration data data in given format to the output out, may be a file, file like object.

  anyconfig.dumps (data, ac_parser=None, **options)

  dumps a configuration data loaded from a string

• Open configuration files:

  anyconfig.open (path, mode=None, ac_parser=None, **options)

  open configuration files with suitable flags and return file/file-like objects, and this object can be passed to the anyconfig.load().

• Merge dicts:

  anyconfig.merge (self, other, ac_merge=MS_DICTS, **options)

  Update (merge) a mapping object ‘self’ with other mapping object ‘other’ or an iterable ‘other’ yields (key, value) tuples according to merge strategy ‘ac_merge’.

• Schema validation and generation of configuration files:

  anyconfig.validate (data, schema, ac_schema_safe=True, ac_schema_errors=False, **options)

  validates configuration data loaded with anyconfig.load() with JSON schema 2 object also loaded with anyconfig.load(). anyconfig.load() may help loading JSON schema file[s] in any formats anyconfig supports.

  anyconfig.gen_schema (data, **options)

  generates a mapping object represents a minimum JSON schema to validate configuration data later. This result object can be serialized to any formats including JSON with anyconfig.dump or anyconfig.dumps.

It enables to load configuration file[s] in various formats in the same manner, and in some cases, even there is no need to take care of the actual format of configuration file[s] like the followings:

import anyconfig

# Config type (format) is automatically detected by filename (file
# extension) in some cases.
conf1 = anyconfig.load("/path/to/foo/conf.d/a.yml")

# Similar to the above but the input is pathlib.Path object.
import pathlib
path_1 = pathlib.Path("/path/to/foo/conf.d/a.yml")
conf1_1 = anyconfig.load(path_1)

# Similar to the first one but load from file object opened:
with anyconfig.open("/path/to/foo/conf.d/a.yml") as fileobj:
    conf1_2 = anyconfig.load(fileobj)

# Loaded config data is a mapping object, for example:
#
#   conf1["a"] => 1
#   conf1["b"]["b1"] => "xyz"
#   conf1["c"]["c1"]["c13"] => [1, 2, 3]

# Or you can specify the format (config type) explicitly if its automatic
# detection may not work.
conf2 = anyconfig.load("/path/to/foo/conf.d/b.conf", ac_parser="yaml")

# Likewise.
with anyconfig.open("/path/to/foo/conf.d/b.conf") as fileobj:
    conf2_2 = anyconfig.load(fileobj, ac_parser="yaml")

# Specify multiple config files by the list of paths. Configurations of each
# files will be merged.
conf3 = anyconfig.load(["/etc/foo.d/a.json", "/etc/foo.d/b.json"])

# Similar to the above but all or one of config file[s] might be missing.
conf4 = anyconfig.load(["/etc/foo.d/a.json", "/etc/foo.d/b.json"],
                       ac_ignore_missing=True)

# Specify config files by glob path pattern:
conf5 = anyconfig.load("/etc/foo.d/*.json")

# Similar to the above, but parameters in the former config file will be simply
# overwritten by the later ones instead of merge:
conf6 = anyconfig.load("/etc/foo.d/*.json", ac_merge=anyconfig.MS_REPLACE)

Also, it can process configuration files which are jinja2-based template files:

• Enables to load a substantial configuration rendered from half-baked configuration template files with given context

• Enables to load a series of configuration files indirectly ‘include’-d from a/some configuration file[s] with using jinja2’s ‘include’ directive.

In [1]: import anyconfig

In [2]: open("/tmp/a.yml", 'w').write("a: {{ a|default('aaa') }}\n")

In [3]: anyconfig.load("/tmp/a.yml", ac_template=True)
Out[3]: {'a': 'aaa'}

In [4]: anyconfig.load("/tmp/a.yml", ac_template=True, ac_context=dict(a='bbb'))
Out[4]: {'a': 'bbb'}

In [5]: open("/tmp/b.yml", 'w').write("{% include 'a.yml' %}\n")  # 'include'

In [6]: anyconfig.load("/tmp/b.yml", ac_template=True, ac_context=dict(a='ccc'))
Out[6]: {'a': 'ccc'}

And python-anyconfig enables to validate configuration files in various formats with using JSON schema like the followings:

# Validate a JSON config file (conf.json) with JSON schema (schema.yaml).
# If validatation succeeds, `rc` -> True, `err` -> ''.
conf1 = anyconfig.load("/path/to/conf.json")
schema1 = anyconfig.load("/path/to/schema.yaml")
(rc, err) = anyconfig.validate(conf1, schema1)  # err is empty if success, rc == 0

# Validate a config file (conf.yml) with JSON schema (schema.yml) while
# loading the config file.
conf2 = anyconfig.load("/a/b/c/conf.yml", ac_schema="/c/d/e/schema.yml")

# Validate config loaded from multiple config files with JSON schema
# (schema.json) while loading them.
conf3 = anyconfig.load("conf.d/*.yml", ac_schema="/c/d/e/schema.json")

# Generate jsonschema object from config files loaded and get string
# representation.
conf4 = anyconfig.load("conf.d/*.yml")
scm4 = anyconfig.gen_schema(conf4)
scm4_s = anyconfig.dumps(scm4, "json")

And you can query loaded data with JMESPath 3 expressions:

In [2]: dic = dict(a=dict(b=[dict(c="C", d=0)]))

In [3]: anyconfig.loads(anyconfig.dumps(dic, ac_parser="json"),
   ...:                 ac_parser="json", ac_query="a.b[0].c")
Out[3]: u'C'

In [4]:

And in the last place, python-anyconfig provides a CLI tool called anyconfig_cli to process configuration files and:

• Convert a/multiple configuration file[s] to another configuration files in different format

• Get configuration value in a/multiple configuration file[s]

• Validate configuration file[s] with JSON schema

• Generate minimum JSON schema file to validate given configuration file[s]

2

http://json-schema.org

3

http://jmespath.org

Supported configuration formats

python-anyconfig supports various file formats if requirements are satisfied and backends in charge are enabled and ready to use:

• Always supported formats of which backends are enabled by default:

Always supported formats

Format	Type	Requirement
JSON	json	json (standard lib) or simplejson 4
Ini-like	ini	configparser (standard lib)
Pickle	pickle	pickle (standard lib)
XML	xml	ElementTree (standard lib)
Java properties 5	properties	None (native implementation with standard lib)
B-sh	shellvars	None (native implementation with standard lib)

• Supported formats of which backends are enabled automatically if requirements are satisfied:

Supported formarts if requirements are satisfied

Format	Type	Requirement
YAML	yaml	ruamel.yaml 6 or PyYAML 7
TOML	toml	toml 8

• Supported formats of which backends are enabled automatically if required plugin modules are installed: python-anyconfig utilizes plugin mechanism provided by setuptools 9 and may support other formats if corresponding plugin backend modules are installed along with python-anyconfig:

Supported formats by pluggable backend modules

Format	Type	Required backend
Amazon Ion	ion	anyconfig-ion-backend 10
BSON	bson	anyconfig-bson-backend 11
CBOR	cbor	anyconfig-cbor-backend 12 or anyconfig-cbor2-backend 13
ConifgObj	configobj	anyconfig-configobj-backend 14
MessagePack	msgpack	anyconfig-msgpack-backend 15

The supported formats of python-anyconfig on your system are able to be listed by ‘anyconfig_cli -L’ like this:

$ anyconfig_cli -L
Supported config types: bson, configobj, ini, json, msgpack, toml, xml, yaml
$

or with the API ‘anyconfig.list_types()’ will show them:

In [8]: anyconfig.list_types()
Out[8]: ['bson', 'configobj', 'ini', 'json', 'msgpack', 'toml', 'xml', 'yaml']

In [9]:

4

https://pypi.python.org/pypi/simplejson

5

ex. https://docs.oracle.com/javase/7/docs/api/java/util/Properties.html

6

https://pypi.python.org/pypi/ruamel.yaml

7

https://pypi.python.org/pypi/PyYAML

8

https://pypi.python.org/pypi/toml

9

http://peak.telecommunity.com/DevCenter/setuptools#dynamic-discovery-of-services-and-plugins

10

https://pypi.python.org/pypi/anyconfig-ion-backend

11

https://pypi.python.org/pypi/anyconfig-bson-backend

12

https://pypi.python.org/pypi/anyconfig-cbor-backend

13

https://pypi.python.org/pypi/anyconfig-cbor2-backend

14

https://pypi.python.org/pypi/anyconfig-configobj-backend

15

https://pypi.python.org/pypi/anyconfig-msgpack-backend

Installation

Requirements

Many runtime dependencies are resolved dynamically and python-anyconfig just disables specific features if required dependencies are not satisfied. Therefore, only python standard library is required to install and use python-anyconfig at minimum.

The following packages need to be installed along with python-anyconfig to enable the features.

Feature	Requirements	Notes
YAML load/dump	ruamel.yaml or PyYAML	ruamel.yaml will be used instead of PyYAML if it’s available to support the YAML 1.2 specification.
TOML load/dump	toml	none
BSON load/dump	bson	bson from pymongo package may work and bson 16 does not
Template config	Jinja2 17	none
Validation with JSON schema	jsonschema 18	Not required to generate JSON schema.
Query with JMESPath expression	jmespath 19	none

16

https://pypi.python.org/pypi/bson/

17

https://pypi.python.org/pypi/Jinja2/

18

https://pypi.python.org/pypi/jsonschema/

19

https://pypi.python.org/pypi/jmespath/

How to install

There is a couple of ways to install python-anyconfig:

• Binary RPMs:

  If you’re running Fedora 27 or later, or CentOS, you can install RPMs from these official yum repos. And if you’re running Red Hat Enterprise Linux 7 or later, you can install RPMs from EPEL repos 20 .

  Or if you want to install the latest version, optionally, you can enable my copr repo, http://copr.fedoraproject.org/coprs/ssato/python-anyconfig/ .

• PyPI: You can install python-anyconfig from PyPI with using pip:

  $ pip install anyconfig
  

• pip from git repo:

  $ pip install git+https://github.com/ssato/python-anyconfig/
  

• Build RPMs from source: It’s easy to build python-anyconfig with using rpm-build and mock:

  # Build Source RPM first and then build it with using mock (better way)
  $ python setup.py bdist_rpm --source-only && mock dist/python-anyconfig-<ver_dist>.src.rpm
  

  or

  # Build Binary RPM to install
  $ python setup.py bdist_rpm
  

  and install RPMs built.

• Build from source: Of course you can build and/or install python modules in usual way such like ‘python setup.py bdist’.

20

Thanks to Brett-san! https://src.fedoraproject.org/rpms/python-anyconfig/

Help and feedbak

If you have any issues / feature request / bug reports with python-anyconfig, please open issue tickets on github.com, https://github.com/ssato/python-anyconfig/issues.

The following areas are still insufficient, I think.

• Make python-anyconfig robust for invalid inputs

• Make python-anyconfig scalable: some functions are limited by max recursion depth.

• Make python-anyconfig run faster: current implementation might be too complex and it run slower than expected as a result.

• Documentation:

  • Especially API docs need more fixes and enhancements! CLI doc is non-fulfilling also.

  • English is not my native lang and there may be many wrong and hard-to-understand expressions.

Any feedbacks, helps, suggestions are welcome! Please open github issues for these kind of problems also!

Hacking

How to test

Run ‘[WITH_COVERAGE=1] ./pkg/runtest.sh [path_to_python_code]’ or ‘tox’ for tests.

About test-time requirements, please take a look at pkg/test_requirements.txt.

How to write backend plugin modules

Backend class must inherit anyconfig.backend.base.Parser or its children in anyconfig.backend.base module and need some members and methods such as load_from_string(), load_from_path(), load_from_stream(), dump_to_string(), dump_to_path() and dump_to_stream(). And anyconfig.backend.tests.ini.Test10 and anyconfig.backend.tests.ini.Test20 may help to write test cases of these methods.

JSON and YAML backend modules (anyconfig.backend.{json,yaml}_) should be good examples to write backend modules and its test cases, I think.

Also, please take a look at some example backend plugin modules mentioned in the Supported configuration formats section.