Usage

Here are some code examples of API usage.

Loading single config file

Here are some example code to load single config file:

import anyconfig

# Config type (format) is automatically detected by filename (file
# extension).
data1 = anyconfig.load("/path/to/foo/conf.d/a.yml")

# Loaded config data is a dict-like object.
# examples:
# data1["a"] => 1
# data1["b"]["b1"] => "xyz"
# data1["c"]["c1"]["c13"] => [1, 2, 3]

# Same as above but I recommend to use the former.
data2 = anyconfig.single_load("/path/to/foo/conf.d/a.yml")

# Or you can specify config type explicitly as needed.
cnf_path = "/path/to/foo/conf.d/b.conf"
data3 = anyconfig.load(cnf_path, ac_parser="yaml")

# Same as above but ...
data4 = anyconfig.single_load(cnf_path, ac_parser="yaml")

# Same as above as a result but make parser instance and pass it explicitly.
yml_psr = anyconfig.find_loader(None, ac_parser="yaml")
data5 = anyconfig.single_load(cnf_path, yml_psr)  # Or: anyconfig.load(...)

# Same as above as a result but make parser instance and pass it explicitly.
yml_psr = anyconfig.find_loader(None, ac_parser="yaml")
data6 = anyconfig.single_load(cnf_path, yml_psr)  # Or: anyconfig.load(...)

# Similar to the previous examples but parser is specified explicitly to use
# ruamel.yaml based YAML parser instead of PyYAML based one, and give
# ruamel.yaml specific option.
data7 = anyconfig.load(cnf_path, ac_parser="ruamel.yaml",
                       allow_duplicate_keys=True)

# Same as above but open the config file explicitly before load.
with anyconfig.open("/path/to/foo/conf.d/a.yml") as istrm:
    data10 = anyconfig.load(istrm)

# Same as above but with specifying config type explicitly.
with anyconfig.open("/path/to/foo/conf.d/a.yml", ac_parser="yaml") as istrm:
    data11 = anyconfig.load(istrm)

Exceptions raised on load

Exception may be raised if something goes wrong. Then, you have to catch them if you want to process more w/ errors ignored or handled.

>>> import anyconfig
>>> anyconfig.single_load(None)
Traceback (most recent call last):
  ...
ValueError: path_or_stream or forced_type must be some value
>>> anyconfig.single_load(None, ac_parser="backend_module_not_avail")
Traceback (most recent call last):
  ...
anyconfig.backends.UnknownParserTypeError: No parser found for type 'backend_module_not_avail'
>>> anyconfig.single_load(None, ac_parser="not_existing_type")
Traceback (most recent call last):
  ...
anyconfig.backends.UnknownParserTypeError: No parser found for type 'not_existing_type'
>>> anyconfig.single_load("unknown_type_file.conf")
Traceback (most recent call last):
  ...
anyconfig.backends.UnknownFileTypeError: No parser found for file 'unknown_type_file.conf'
>>>

Common and backend specific Keyword options on load single config file

Here is a brief summary of keyword options prefixed with ‘ac_’ to change the behavior on load.

Option

Type

Note

ac_parser

str or anyconfig.backend.base.Parser

Forced parser type or parser object

ac_dict

callable

Any callable (function or class) to make mapping object will be returned as a result or None. If not given or ac_dict is None, default mapping object used to store resutls is dict or OrderedDict if ac_ordered is True and selected backend can keep the order of items in mapping objects.

ac_ordered

bool

True to keep resuls ordered. Please note that order of items in results may be lost depends on backend used.

ac_template

bool

Assume given file may be a template file and try to compile it AAR if True

ac_context

mapping object

Mapping object presents context to instantiate template

ac_schema

str

JSON schema file path to validate given config file

ac_query

str

JMESPath expression to query data

You can pass backend (config loader) specific keyword options to these load and dump functions as needed along with the above anyconfig specific keyword options:

# from python -c "import json; help(json.load)":
# Help on function load in module json:
#
# load(fp, encoding=None, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw)
#    Deserialize ``fp`` (a ``.read()``-supporting file-like object containing
#    a JSON document) to a Python object.
#    ...
data6 = anyconfig.load("foo.json", parse_float=None)

Allowed keyword options depend on backend, so please take a look at each backend API docs for more details about it.

Others topics on load

Anyconfig also enables:

Note

The returned object is a mapping object, dict or collections.OrderedDict object by default.

Loading multiple config files

Here are some example code to load multiple config files:

import anyconfig

# Specify config files by list of paths:
data1 = anyconfig.load(["/etc/foo.d/a.json", "/etc/foo.d/b.json"])

# Similar to the above but all or one of config files are missing:
data2 = anyconfig.load(["/etc/foo.d/a.json", "/etc/foo.d/b.json"],
                       ignore_missing=True)

# Specify config files by glob path pattern:
cnf_path = "/etc/foo.d/*.json"
data3 = anyconfig.load(cnf_path)

# Similar to above but make parser instance and pass it explicitly.
psr = anyconfig.find_loader(cnf_path)
data4 = anyconfig.load(cnf_path, psr)

# Similar to the above but parameters in the former config file will be simply
# overwritten by the later ones:
data5 = anyconfig.load("/etc/foo.d/*.json", ac_merge=anyconfig.MS_REPLACE)

Strategies to merge data loaded from multiple config files

On loading multiple config files, you can choose ‘strategy’ to merge configurations from the followings and pass it with ac_merge keyword option:

  • anyconfig.MS_REPLACE: Replace all configuration parameter values provided in former config files are simply replaced w/ the ones in later config files.

    For example, if a.yml and b.yml are like followings:

    a.yml:

    a: 1
b:
   - c: 0
   - c: 2
d:
   e: "aaa"
   f: 3

    b.yml:

    b:
   - c: 3
d:
   e: "bbb"

    then:

    load(["a.yml", "b.yml"], ac_merge=anyconfig.MS_REPLACE)

    will give object such like:

    {'a': 1, 'b': [{'c': 3}], 'd': {'e': "bbb"}}

  • anyconfig.MS_NO_REPLACE: Do not replace configuration parameter values provided in former config files.

    For example, if a.yml and b.yml are like followings:

    a.yml:

    b:
   - c: 0
   - c: 2
d:
   e: "aaa"
   f: 3

    b.yml:

    a: 1
b:
   - c: 3
d:
   e: "bbb"

    then:

    load(["a.yml", "b.yml"], ac_merge=anyconfig.MS_NO_REPLACE)

    will give object such like:

    {'a': 1, 'b': [{'c': 0}, {'c': 2}], 'd': {'e': "bbb", 'f': 3}}

  • anyconfig.MS_DICTS (default): Merge dicts recursively. That is, the following:

    load(["a.yml", "b.yml"], ac_merge=anyconfig.MS_DICTS)

    will give object such like:

    {'a': 1, 'b': [{'c': 3}], 'd': {'e': "bbb", 'f': 3}}

    This is the merge strategy chosen by default.

  • anyconfig.MS_DICTS_AND_LISTS: Merge dicts and lists recursively. That is, the following:

    load(["a.yml", "b.yml"], ac_merge=anyconfig.MS_DICTS_AND_LISTS)

    will give object such like:

    {'a': 1, 'b': [{'c': 0}, {'c': 2}, {'c': 3}], 'd': {'e': "bbb", 'f': 3}}

Or you you can implement custom function or class or anything callables to merge nested dicts by yourself and utilize it with ac_merge keyword option like this:

def my_merge_fn(self, other, key, val=None, **options):
    """
    :param self: mapping object to update with `other`
    :param other: mapping object to update `self`
    :param key: key of mapping object to update
    :param val: value to update self alternatively

    :return: None but `self` will be updated
    """
    if key not in self:
        self[key] = other[key] if val is None else val

load(["a.yml", "b.yml"], ac_merge=my_merge_fn)

Please refer to the existing functions in anyconfig.dicsts (_update_* functions) to implement custom functions to merge nested dicts for more details.

Common and backend specific Keyword options on load multiple config files

Here is a brief summary of keyword options prefixed with ‘ac_’ in addition to the keyword options explained in the Common and backend specific Keyword options on load single config file section to change the behavior on load multiple files.

Option

Type

Note

ac_merge

str

One of anyconfig.dicts.MERGE_STRATEGIES to select strategy of how to merge results loaded from multiple configuration files. See the doc of anyconfig.dicts for more details of strategies. The default is anyconfig.dicts.MS_DICTS.

ac_marker

str

Glob marker string to detect paths patterns. ‘*’ by default.

Dumping config data

A pair of APIs are provided to dump config data loaded w/ using loading APIs as described previously and corresponding to them.

  • dumps(): Dump data as a string

  • dump(): Dump data to file of which path was given or file-like object opened

Note

To specify the format or backend type w/ ac_parser keyword option is necessary for dumps() API because anyconfig cannot determine the type w/o it.

Like loading APIs, you can pass common and backend specific keyword options to them.

  • common keyword options: ac_parser to determine which backend to use

  • backend specific keyword options: see each backends’ details

Here are some examples of these usage:

In [1]: s = """a: A
   .....: b:
   .....:   - b0: 0
   .....:   - b1: 1
   .....: c:
   .....:   d:
   .....:     e: E
   .....: """

In [2]: cnf = anyconfig.loads(s, ac_parser="yaml")

In [3]: cnf
Out[3]: {'a': 'A', 'b': [{'b0': 0}, {'b1': 1}], 'c': {'d': {'e': 'E'}}}

In [4]: anyconfig.dumps(cnf, ac_parser="yaml")  # ac_parser option is necessary.
Out[4]: 'a: A\nc:\n  d: {e: E}\nb:\n- {b0: 0}\n- {b1: 1}\n'

In [5]: print(anyconfig.dumps(cnf, ac_parser="yaml"))
a: A
c:
  d: {e: E}
b:
- {b0: 0}
- {b1: 1}

In [6]: print(anyconfig.dumps(cnf, ac_parser="json"))
{"a": "A", "c": {"d": {"e": "E"}}, "b": [{"b0": 0}, {"b1": 1}]}

In [7]: print(anyconfig.dumps(cnf, ac_parser="ini"))  # It cannot!
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-228-2b2771a44a7e> in <module>()
----> 1 print(anyconfig.dumps(cnf, ac_parser="ini"))
    ...
AttributeError: 'str' object has no attribute 'iteritems'

In [8]: print(anyconfig.dumps(cnf, ac_parser="configobj"))
a = A
b = {'b0': 0}, {'b1': 1}
[c]
[[d]]
e = E

In [9]:

Like this example, it’s not always possible to dump data to any formats because of limitations of formats and/or backends.

Keep the order of configuration items

If you want to keep the order of configuration items, specify ac_order=True on load or specify ac_dict to any mapping object can save the order of items such like collections.OrderedDict (or OrderedDict). Otherwise, the order of configuration items will be lost by default.

Please note that anyconfig.load APIs sometimes cannot keep the order of items in the original data even if ac_order=True or ac_dict=<ordereddict> was specified because used backend or module cannot keep that. For example, JSON backend can keep items but current YAML backend does not due to the limitation of YAML module it using.

Validation with and/or generate JSON Schema

If jsonschema [4] is installed and available, you can validate config files with using anyconfig.validate() since 0.0.10.

# Validate a JSON config file (conf.json) with JSON schema (schema.json).
# If validation succeeds, `rc` -> True, `err` -> ''.
conf1 = anyconfig.load("/path/to/conf.json")
schema1 = anyconfig.load("/path/to/schema.json")
(rc, err) = anyconfig.validate(conf1, schema1)

# Similar to the above but both config and schema files are in YAML.
conf2 = anyconfig.load("/path/to/conf.yml")
schema2 = anyconfig.load("/path/to/schema.yml")
(rc, err) = anyconfig.validate(conf2, schema2)

# Similar to the above but exception will be raised if validation fails.
(rc, _err) = anyconfig.validate(conf2, schema2, ac_schema_safe=False)

It’s also able to validate config files during load:

# Validate a config file (conf.yml) with JSON schema (schema.yml) while
# loading the config file.
conf1 = anyconfig.load("/a/b/c/conf.yml", ac_schema="/c/d/e/schema.yml")

# Validate config loaded from multiple config files with JSON schema
# (schema.json) while loading them.
conf2 = anyconfig.load("conf.d/*.yml", ac_schema="/c/d/e/schema.json")

And even if you don’t have any JSON schema files, don’t worry ;-), anyconfig can generate the schema for your config files on demand and you can save it in any formats anyconfig supports.

# Generate a simple JSON schema file from config file loaded.
conf1 = anyconfig.load("/path/to/conf1.json")
schema1 = anyconfig.gen_schema(conf1)
anyconfig.dump(schema1, "/path/to/schema1.yml")

# Generate more strict (precise) JSON schema file from config file loaded.
schema2 = anyconfig.gen_schema(conf1, ac_schema_strict=True)
anyconfig.dump(schema2, "/path/to/schema2.json")

Note

If you just want to generate JSON schema from your config files, then you don’t need to install jsonschema in advance because anyconfig can generate JSON schema without jsonschema module.

Template config support

anyconfig supports template config files since 0.0.6. That is, config files written in Jinja2 template [5] will be compiled before loading w/ backend module.

Note

Template config support is disabled by default to avoid side effects when processing config files of jinja2 template or having some expressions similar to jinaj2 template syntax.

Anyway, a picture is worth a thousand words. Here is an example of template config files.

ssato@localhost% cat a.yml
a: 1
b:
  {% for i in [1, 2, 3] -%}
  - index: {{ i }}
  {% endfor %}
{% include "b.yml" %}
ssato@localhost% cat b.yml
c:
  d: "efg"
ssato@localhost% anyconfig_cli a.yml --template -O yaml -s
a: 1
b:
- {index: 1}
- {index: 2}
- {index: 3}
c: {d: efg}
ssato@localhost%

And another one:

In [1]: import anyconfig

In [2]: ls *.yml
a.yml  b.yml

In [3]: cat a.yml
a: {{ a }}
b:
  {% for i in b -%}
  - index: {{ i }}
  {% endfor %}
{% include "b.yml" %}

In [4]: cat b.yml
c:
  d: "efg"

In [5]: context = dict(a=1, b=[2, 4])

In [6]: anyconfig.load("*.yml", ac_template=True, ac_context=context)
Out[6]: {'a': 1, 'b': [{'index': 2}, {'index': 4}], 'c': {'d': 'efg'}}

Query results with JMESPath expression

anyconfig supports to query result mapping object with JMESPath expression since 0.8.3 like the following example [6] .

>>> yaml_s = """\
... locations:
...   - name: Seattle
...     state: WA
...   - name: New York
...     state: NY
...   - name: Olympia
...     state: WA
... """
>>> query = "locations[?state == 'WA'].name | sort(@) | {WashingtonCities: join(', ', @)}"
>>> anyconfig.loads(yaml_s, ac_parser="yaml", ac_query=query)
 {'WashingtonCities': 'Olympia, Seattle'}
>>>

Different from other libraries can process JMESPath expressions, anyconfig can query data of any formats it supports, with help of the jmespath support library [7] . That is, you can query XML, YAML, BSON, Toml, and, of course JSON files with JMESPath expression.

Other random topics with API usage

Suppress logging messages from anyconfig module

anyconfig uses a global logger named anyconfig and logging messages are suppressed by default as NullHandler was attached to the logger [8] . If you want to see its log messages out, you have to configure it (add handler and optionally set log level) like the followings.

  • Set the log level and handler of anyconfig module before load to print log messages such as some backend modules are not available, when it’s initialized:

In [1]: import logging

In [2]: LOGGER = logging.getLogger("anyconfig")

In [3]: LOGGER.addHandler(logging.StreamHandler())

In [4]: LOGGER.setLevel(logging.ERROR)

In [5]: import anyconfig

In [6]: anyconfig.dumps(dict(a=1, b=[1,2]), "aaa")
No parser found for given type: aaa
Out[6]: '{"a": 1, "b": [1, 2]}'

In [7]:

  • Set log level of anyconfig module after load:

In [1]: import anyconfig, logging

In [2]: LOGGER = logging.getLogger("anyconfig")

In [3]: LOGGER.addHandler(logging.StreamHandler())

In [4]: anyconfig.dumps(dict(a=2, b=[1,2]), "unknown_type")
No parser found for given type: unknown_type
Parser unknown_type was not found!
Dump method not implemented. Fallback to json.Parser
Out[4]: '{"a": 2, "b": [1, 2]}'

In [5]:

Combination with other modules

anyconfig can be combined with other modules such as pyxdg and appdirs [9] .

For example, you can utilize anyconfig and pyxdg or appdirs in you application software to load user config files like this:

import anyconfig
import appdirs
import os.path
import xdg.BaseDirectory

APP_NAME = "foo"
APP_CONF_PATTERN = "*.yml"


def config_path_by_xdg(app=APP_NAME, pattern=APP_CONF_PATTERN):
    return os.path.join(xdg.BaseDirectory.save_config_path(app), pattern)


def config_path_by_appdirs(app=APP_NAME, pattern=APP_CONF_PATTERN):
    os.path.join(appdirs.user_config_dir(app), pattern)


def load_config(fun=config_path_by_xdg):
    return anyconfig.load(fun())

Default config values

Current implementation of anyconfig.*load*() do not provide a way to provide some sane default configuration values (as a dict parameter for example) before/while loading config files. Instead, you can accomplish that by a few lines of code like the followings:

import anyconfig

conf = dict(foo=0, bar='1', baz=[2, 3])  # Default values
conf_from_files = anyconfig.load("/path/to/config_files_dir/*.yml")
anyconfig.merge(conf, conf_from_files)  # conf will be updated.

# Use `conf` ...

or:

conf = dict(foo=0, bar='1', baz=[2, 3])
anyconfig.merge(conf, anyconfig.load("/path/to/config_files_dir/*.yml"))

Environment Variables

It’s a piece of cake to use environment variables as config default values like this:

conf = os.environ.copy()
anyconfig.merge(conf, anyconfig.load("/path/to/config_files_dir/*.yml"))

Load from compressed files

Since 0.2.0, python-anyconfig can load configuration from file or file-like object, called stream internally. And this should help loading configurations from compressed files.

  • Loading from a compressed JSON config file:

import gzip

strm = gzip.open("/path/to/gzip/compressed/cnf.json.gz")
cnf = anyconfig.load(strm, "json")

  • Loading from some compressed JSON config files:

import gzip
import glob

cnfs = "/path/to/gzip/conf/files/*.yml.gz"
strms = [gzip.open(f) for f in sorted(glob.glob(cnfs))]
cnf = anyconfig.load(strms, "yaml")

Please note that “json” argument passed to anyconfig.load is necessary to help anyconfig find out the configuration type of the file.

Convert from/to bunch objects

It’s easy to convert result conf object from/to bunch objects [10] as anyconfig.load{s,} return a dict-like object:

import anyconfig
import bunch

conf = anyconfig.load("/path/to/some/config/files/*.yml")
bconf = bunch.bunchify(conf)
bconf.akey = ...  # Overwrite a config parameter.
   ...
anyconfig.dump(bconf.toDict(), "/tmp/all.yml")