英文版: Tips that may save you from the hell of PyYAML

YAML 是一个被广泛使用的数据序列化和配置语言，作为一个开发者，总是不免和它打交道。 但处理 YAML 文档，尤其是使用 PyYAML 的过程总是非常痛苦。

这篇文章分享我在 Python 下使用 PyYAML 的技巧和代码片段，并介绍几个相关的库。

注意：本文中的代码仅保证在 Python 3 下正常工作

总是使用 safe_load/safe_dump

PyYAML 的 load 函数可以构造任意 Python 对象（Pickle 协议），这意味着一次 load 可能导致任意 Python 函数被执行。

为了确保应用程序的安全性，尽量在任何情况下使用 yaml.safe_load 和 yaml.safe_dump。

保留字段顺序

Python 3.7+ 中，dict keys 具备保留插入顺序的特性，所以通过 yaml.safe_load 得到的 dict，其 keys 顺序会与原始文件保持一致。

>>> import yaml
>>> text = """---
... c: 1
... b: 1
... d: 1
... a: 1
... """
>>> d = yaml.safe_load(text)
>>> d
{'c': 1, 'b': 1, 'd': 1, 'a': 1}
>>> list(d)
['c', 'b', 'd', 'a']

当把 dict 导出为 YAML 字符串时，为 yaml.safe_dump 传递 sort_keys=False 来保留 keys 的顺序。

>>> print(yaml.safe_dump(d))
a: 1
b: 1
c: 1
d: 1
>>> d['e'] = 1
>>> print(yaml.safe_dump(d, sort_keys=False))
c: 1
b: 1
d: 1
a: 1
e: 1

如果 Python 版本较低，或者你想确保代码能在更广泛的环境下工作，你可以使用 oyaml 库来代替 PyYAML 的 yaml 包。

>>> import oyaml as yaml
>>> d = yaml.safe_load(text)
>>> d
OrderedDict([('c', 1), ('b', 1), ('d', 1), ('a', 1)])
>>> d['e'] = 1
>>> print(yaml.safe_dump(d, sort_keys=False))
c: 1
b: 1
d: 1
a: 1
e: 1

优化列表项的缩进

默认情况下，PyYAML 输出的列表缩进与其父元素一致。

>>> d = {'a': [1, 2, 3]}
>>> print(yaml.safe_dump(d))
a:
- 1
- 2
- 3

这并不是很好的格式，根据 Ansible 和 HomeAssistant 等 YAML 书写规范，列表项应该缩进 2 空格。

这种格式也会对导致列表项不会被如 VSCode 等编辑器识别，进而无法使用编辑器的折叠功能。

要解决这个问题，使用如下代码片段，在代码中定义 IndentDumper class:

class IndentDumper(yaml.Dumper):
    def increase_indent(self, flow=False, indentless=False):
        return super(IndentDumper, self).increase_indent(flow, False)

然后将它传递给 yaml.dump 的 Dumper 关键字参数。

>>> print(yaml.dump(d, Dumper=IndentDumper))
a:
  - 1
  - 2
  - 3

注意，yaml.safe_dump 由于有自己的 Dumper class，传递此参数会造成冲突。

输出可读的 UTF-8 字符

默认情况下，PyYAML 假设你希望输出的结果里只有 ASCII 字符。

>>> d = {'a': '你好'}
>>> print(yaml.safe_dump(d))
a: "\u4F60\u597D"

这会让输出结果非常难以阅读。

在 UTF-8 足够普及的今天，直接输出 UTF-8 字符是非常安全的。 因此我们可以将 allow_unicode=True 传入 yaml.safe_dump 使 PyYAML 将 Unicode 转换成 UTF-8 字符串。

>>> print(yaml.safe_dump(d, allow_unicode=True))
a: 你好

一些 YAML 相关的库

oyaml

Link: https://github.com/wimglenn/oyaml

正如上文中提到的，oyaml 是 yaml 包的替换品，使 dict keys 的顺序在 dump/load 的时候得以保留。

oyaml 是一个单文件库，只有 53 行代码，因此使用起来非常灵活，你可以直接把它的代码复制到自己的项目中，然后根据自己的需求进行修改。

strictyaml

Link: https://github.com/crdoconnor/strictyaml

有的人说 YAML 过于复杂和灵活，不是一个好的配置语言。但我认为这不是 YAML 的问题，而是使用方式的问题。如果我们限制程序只使用 YAML 的部分功能，YAML 其实可以变得像它设计的那般好用。

这就是 StrictYAML 的设计意图，它是一个类型安全的 YAML 解析器，实现了 YAML 规范说明中的一个子集 。

如果你对 YAML 的输入输出有较强的安全考虑，建议使用 StrictYAML 代替 PyYAML。

顺带一提的是，StrictYAML 的文档站有很多关于设计细节和配置语言思考的文章，非常值得一看。

ruamel.yaml

Link: https://yaml.readthedocs.io/en/latest/overview.html

ruamel.yaml 是 PyYAML 的一个分叉，于 2009 年发布并持续维护至今。

ruamel.yaml 的文档里详细说明了它和 PyYAML 的差异。 总体来说，ruamel.yaml 专注在 YAML 1.2 上，对一些语法进行了优化。

ruamel.yaml 最令我感兴趣的特性是输入输出的 “round-trip”，可以最大程度地保留输入源的原始格式。官方文档中的定义是这样的：

A round-trip is a YAML load-modify-save sequence and ruamel.yaml tries to preserve, among others:

• comments
• block style and key ordering are kept, so you can diff the round-tripped source
• flow style sequences ( ‘a: b, c, d’) (based on request and test by Anthony Sottile)
• anchor names that are hand-crafted (i.e. not of the formidNNN)
• merges in dictionaries are preserved

如果你有尽可能保留原始格式的需求，建议使用 ruamel.yaml 代替 PyYAML。

在使用中我注意到 ruamel.yaml 的 safe load 方法 (YAML(typ='safe').load) 与 PyYAML 有些不同，它无法解析 flow style 的集合定义 (如 a: {"foo": "bar"})，这点没有在文档中提及，使用时须多加注意。

总结

YAML 有它好的地方和坏的地方。它易于阅读，初期的学习曲线非常平缓。 但 YAML 的规范说明非常复杂，不仅造成了使用中的混乱，也使不同语言的实现在很多细微的地方难以保持一致。

尽管有这些小毛病，YAML 仍然是我心中最好的配置语言。希望这篇文章所介绍的技巧能够帮助你避免问题，获得更好的开发和使用体验。

Code in this article is only guaranteed to work in Python 3

Always use safe_load/safe_dump

YAML’s ability to construct an arbitrary Python object makes it dangerous to use blindly. It might be harmful to your application to simply yaml.load a document from an untrusted source such as the Internet and user input.

See from PyYAML official documentation:

Warning: It is not safe to call yaml.load with any data received from an untrusted source! yaml.load is as powerful as pickle.load and so may call any Python function.

In short, you should always use yaml.safe_load and yaml.safe_dump as the standard I/O methods for YAML.

Keep keys in order (load/dump)

In Python 3.7+, the order of dict keys is naturally preserved ¹, thus the dict you get from yaml.safe_load has the same order of keys as the original file.

>>> import yaml
>>> text = """---
... c: 1
... b: 1
... d: 1
... a: 1
... """
>>> d = yaml.safe_load(text)
>>> d
{'c': 1, 'b': 1, 'd': 1, 'a': 1}
>>> list(d)
['c', 'b', 'd', 'a']

When dumping dict into a YAML string, make sure to add keyword argument sort_keys=False to preserve the order of keys.

>>> print(yaml.safe_dump(d))
a: 1
b: 1
c: 1
d: 1
>>> d['e'] = 1
>>> print(yaml.safe_dump(d, sort_keys=False))
c: 1
b: 1
d: 1
a: 1
e: 1

If your Python version is lower, or you want to make sure the keys order preserving always works, you can use this library called oyaml as a drop-in replacement for pyyaml.

>>> import oyaml as yaml
>>> d = yaml.safe_load(text)
>>> d
OrderedDict([('c', 1), ('b', 1), ('d', 1), ('a', 1)])
>>> d['e'] = 1
>>> print(yaml.safe_dump(d, sort_keys=False))
c: 1
b: 1
d: 1
a: 1
e: 1

Enhance list indentation (dump)

By default, PyYAML indent list items on the same level as their parent.

>>> d = {'a': [1, 2, 3]}
>>> print(yaml.safe_dump(d))
a:
- 1
- 2
- 3

This is not a good format according to style guides like Ansible and HomeAssistant. It is also not recognized by code editors like VSCode, making the list items unfoldable in the editor.

To solve this problem, you can use the snippet below to define an IndentDumper class:

class IndentDumper(yaml.Dumper):
    def increase_indent(self, flow=False, indentless=False):
        return super(IndentDumper, self).increase_indent(flow, False)

Then pass it to the Dumper keyword argument in yaml.dump function.

>>> print(yaml.dump(d, Dumper=IndentDumper))
a:
  - 1
  - 2
  - 3

Note that Dumper cannot be passed to yaml.safe_dump which has its owner dumper class defined.

Output readable UTF-8 (dump)

By default, PyYAML assumes the user only wants ASCII code in the output, so it converts UTF-8 characters to Python’s Unicode representation.

>>> d = {'a': '你好'}
>>> print(yaml.safe_dump(d))
a: "\u4F60\u597D"

This makes the output hard to read for humans.

In the modern world, UTF-8 is widely supported, it’s safe to write UTF-8 in the output. Pass allow_unicode=True to yaml.safe_dump to enable that.

>>> print(yaml.safe_dump(d, allow_unicode=True))
a: 你好

No default_flow_style needed (dump)

Most of the time we don’t want flow style productions in the output (i.e. no JSON in YAML). According to PyYAML documentation, default_flow_style=False should be passed to yaml.safe_dump to achieve that.

After digging into the source code of the latest PyYaml (6.0), I find it is not needed anymore. You should remove this keyword argument to keep the code cleaner and less confusing.

Libraries

oyaml

Link: https://github.com/wimglenn/oyaml

As mentioned above, oyaml is a drop-in replacement for PyYAML which preserves dict ordering.

I suggest using oyaml if you already use PyYAML in your code.

It’s worth mentioning that oyaml is a single-file library with only 53 lines of code. This makes it very flexible to use, you can just copy the code to your library and customize it according to your need.

strictyaml

Link: https://github.com/crdoconnor/strictyaml

Some people say YAML is too complex and flexible to be a good configuration language, but I think this is not the problem of YAML, but the problem of how we use it. If we restrict our usage to only a subset of its features, it will be as good as it should be.

This is where StrictYAML came up. It is a type-safe YAML parser that parses and validates a restricted subset of the YAML specification.

I suggest using StrictYAML if you have strong security concerns for your application.

There are tons of great articles on the documentation site of strictyaml, definitely worth having a look at if you have thought about YAML and other configuration languages.

ruamel. yaml

Link: https://yaml.readthedocs.io/en/latest/overview.html

ruamel.yaml is a fork of PyYAML, it was released in 2009 and continuously maintained in the past decade.

The differences with PyYAML are listed here. Generally, ruamel.yaml focuses on YAML 1.2 with some opinionated enhancements for the syntax.

What interests me most is the ability to round-trip in the loading/dumping process. It works like black magic. Here’s the explanation from ruamel.yaml documentation:

A round-trip is a YAML load-modify-save sequence and ruamel.yaml tries to preserve, among others:

• comments
• block style and key ordering are kept, so you can diff the round-tripped source
• flow style sequences ( ‘a: b, c, d’) (based on request and test by Anthony Sottile)
• anchor names that are hand-crafted (i.e. not of the formidNNN)
• merges in dictionaries are preserved

I suggest using ruamel.yaml if you have the requirement to preserve the original content as much as possible.

One thing I notice is that ruamel.yaml’s safe_load method (YAML(typ='safe').load) cannot parse flow style collection (a: {"foo": "bar"}), this is a undocumented difference with PyYAML.

Summary

YAML has its good and bad. It’s easy to read, the learning curve is mild at the beginning, but the specification is complex, which not only causes chaos in practice, but also makes implementations in different languages inconsistent with each other in many trivial aspects.

Despite these quirks, YAML is still the best configuration language for me, and as long as we can use it properly, problems will be avoided and the experience will be much better.