tests: Write tags as YAML lists
Twister no longer splits space-separated lists, and now rejects them for keys that name single items, such as tags. Any twister run that scans these tests fails to load them:
ERROR - …/tests/unit/test1_unit_tests: can’t load (skipping): ConfigurationError(“…/tests/unit/test1_unit_tests/testcase.yaml: whitespace in ‘tags’ value ‘zcbor unit’ in test ‘zcbor.unit.test1’. Space-separated lists are not supported, write the value as a YAML list instead”)
Upstream zcbor made the same change in 6212e824d7ab (“tests: Update testcase.yaml files to not use space-separated lists”); this applies the tags part of it here, as platform_allow was already converted.
Fixes: zephyrproject-rtos/zephyr#121467 Signed-off-by: Otavio Salvador otavio@ossystems.com.br
版权所有:中国计算机学会技术支持:开源发展技术委员会
京ICP备13000930号-9
京公网安备 11010802047560号
zcbor
zcbor is a low footprint CBOR library in the C language (C++ compatible), tailored for use in microcontrollers. It comes with a schema-driven script tool that can validate your data, or even generate code. The schema language (CDDL) allows creating very advanced and detailed schemas.
The validation and conversion part of the tool works with YAML and JSON data, in addition to CBOR. It can for example validate a YAML file against a schema and convert it into CBOR.
The code generation part of the tool generates C code based on the given schema. The generated code performs CBOR encoding and decoding using the C library, while also validating the data against all the rules in the schema.
The schema language used by zcbor is CDDL (Concise Data Definition Language) which is a powerful human-readable data description language defined in IETF RFC 8610.
Features
Here are some possible ways zcbor can be used:
Getting started
There are samples in the samples directory that demonstrate different ways to use zcbor, both the script tool and the C code.
The tests also demonstrate how to use zcbor in different ways. The encoding, decoding, and unit tests run using Zephyr (the samples do not use Zephyr).
Should I use code generation or the library directly?
The benefit of using code generation is greater for decoding than encoding. This is because decoding is generally more complex than encoding, since when decoding you have to gracefully handle all possible payloads. The code generation will provide a number of checks that are tedious to write manually. These checks ensure that the payload is well-formed.
CBOR decoding/encoding library
The CBOR library can be found in include/ and src/ and can be used directly, by including the files in your project. If using zcbor with Zephyr, the library will be available when the CONFIG_ZCBOR config is enabled.
The library is also used by generated code. See the Code generation section for more info about code generation.
The C library is C++ compatible.
The zcbor state object
To do encoding or decoding with the library, instantiate a
zcbor_state_tobject, which is most easily done using theZCBOR_STATE_*()macros, look below or in the hello_world sample for example code.The
elem_countmember refers to the number of encoded objects in the current list or map.elem_countstarts again when entering a nested list or map, and is restored when exiting.elem_countis one reason for needing “backup” states (the other is to allow rollback of the payload). Backups are needed for decoding if there are any lists, maps, or CBOR-encoded strings (zcbor_bstr_*_decode) in the data. Backups are needed for encoding if there are any lists or maps and you are using canonical encoding (ZCBOR_CANONICAL), or when using thezcbor_bstr_*_encodefunctions.n_flagsis used when decoding maps where the order is unknown. It allows using thezcbor_unordered_map_search()function to search for elements.See the header files for more information.
Configuration
The C library has a few compile-time configuration options. These configuration options can be enabled by adding them as compile definitions to the build. If using zcbor with Zephyr, use the Kconfig options instead.
ZCBOR_CANONICALZCBOR_CANONICALincreases code size and makes the encoding library more often use state backups. When decoding, ifenforce_canonicalis true, ensure that the incoming data conforms to canonical encoding, i.e. no indefinite length encoding, and always using minimal length encoding (e.g. not using 16 bits to encode a value < 256). EnablingZCBOR_CANONICALchanges the default ofenforce_canonicalfromfalsetotrueNote: the map ordering constraint in canonical encoding is not checked.ZCBOR_VERBOSEzcbor_log()), and also a trace message (zcbor_trace()) for each decoded value, and in each generated function (when using code generation).ZCBOR_ASSERTSzcbor_assert()). When they fail, the assert statements instruct the current function to return aZCBOR_ERR_ASSERTIONerror. IfZCBOR_VERBOSEis enabled, a message is printed.ZCBOR_STOP_ON_ERRORstop_on_errorfunctionality. Note that it also has to be enabled in the state variable (state->constant_state->stop_on_error). This makes all zcbor functions abort their execution if called when an error has already happened.ZCBOR_BIG_ENDIANZCBOR_MAP_SMART_SEARCHPython script and module
The zcbor.py script can directly read CBOR, YAML, or JSON data and validate it against a CDDL description. It can also freely convert the data between CBOR/YAML/JSON. It can also output the data to a C file formatted as a byte array.
Invoking zcbor.py from the command line
zcbor.py can be installed via
pip, or alternatively invoked directly from its location in this repo.Following are some generalized examples for validating, and for converting (which also validates) data from the command line. The script infers the data format from the file extension, but the format can also be specified explicitly. See
zcbor validate --helpandzcbor convert --helpfor more information.Or directly from within the repo.
Importing zcbor in a Python script
Importing zcbor gives access to the DataTranslator class which is used to implement the command line conversion features. DataTranslator can be used to programmatically perform the translations, or to manipulate the data. When accessing the data, you can choose between two internal formats:
Making CBOR YAML-/JSON-compatible
Since CBOR supports more data types than YAML and JSON, zcbor can optionally use a bespoke format when converting to/from YAML/JSON. This is controlled with the
--yaml-compatibilityoption toconvertandvalidate. This is relevant when handling YAML/JSON conversions of data that uses the unsupported features. The following data types are supported by CBOR, but not by YAML (or JSON which is a subset of YAML):{"zcbor_bstr": "<hex-formatted bytestring>"}, or as{"zcbor_bstr": <any type>}if the CBOR bytestring contains CBOR-formatted data, in which the data is decoded into<any type>.{"zcbor_keyval<unique int>": {"key": <key, not text>, "val": <value>}}.cbor2.CBORTag. In YAML, these are represented as{"zcbor_tag": <tag number>, "zcbor_tag_val": <tagged data>}.cbor2.types.undefined. In YAML, undefined is represented as:["zcbor_undefined"].You can see an example of the conversions in tests/cases/yaml_compatibility.yaml and its CDDL file tests/cases/yaml_compatibility.cddl.
Code generation
Code generation is invoked with the
zcbor codecommand:When you call this, zcbor reads the CDDL files and creates C struct types to match the types described in the CDDL. It then creates code that uses the C library to decode CBOR data into the structs, and/or encode CBOR from the data in the structs. Finally, it takes the “entry types” (
-t) and creates a public API function for each of them. While doing these things, it will make a number of optimizations, e.g. inlining code for small types and removing unused functions. It outputs the generated code into header and source files and optionally creates a CMake file to build them.The
zcbor codecommand reads one or more CDDL file(s) and generates some or all of these files:--decode/-dis specified)--decode/-dis specified)--encode/-eis specified)--encode/-eis specified)--output-cmakeis specified).CDDL allows placing restrictions on the members of your data. Restrictions can be on type (int/string/list/bool etc.), on content (e.g. values/sizes of ints or strings), and repetition (e.g. the number of members in a list). The generated code will validate the input, which means that it will check all the restriction set in the CDDL description, and fail if a restriction is broken.
There are tests for the code generation in tests/decode and tests/encode. The tests require Zephyr (if your system is set up to build Zephyr samples, the tests should also build).
The generated C code is C++ compatible.
Build system
When calling zcbor with the argument
--output-cmake <file path>, a CMake file will be created at that location. The generated CMake file creates a target library and adds the generated and non-generated source files as well as required include directories to it. This CMake file can then be included in your project’sCMakeLists.txtfile, and the target can be linked into your project. This is demonstrated in the tests, e.g. at tests/decode/test3_simple/CMakeLists.txt. zcbor can be instructed to copy the non-generated sources to the same location as the generated sources with--copy-sources.Usage Example
There are buildable examples in the samples directory.
To see how to use the C library directly, see the hello_world sample, or the pet sample (look for calls to functions prefixed with
zcbor_).To see how to use code generation, see the pet sample.
Look at the CMakeLists.txt file to see how zcbor is invoked for code generation (and for conversion).
To see how to do conversion, see the pet sample.
Below are some additional examples of how to invoke zcbor for code generation and for converting/validating
Code generation
Converting
Here is an example call for converting from YAML to CBOR:
Which takes a yaml structure from mypet.yaml, validates it against the Pet type in the CDDL description in pet.cddl, and writes binary CBOR data to mypet.cbor.
Validating
Here is an example call for validating a JSON file:
Which takes the json structure in mypet.json, converts any yaml-compatible values to their original form, and validates that against the Pet type in the CDDL description in pet.cddl.
Running tests
The tests for the generated code are based on the Zephyr ztest library. These tests can be found in tests/decode and tests/encode. To set up the environment to run the ztest tests, follow Zephyr’s Getting Started Guide, or see the workflow in the
.githubdirectory.Tests for
convertandverifyare implemented with the unittest module. These tests can be found in tests/scripts/test_zcbor.py. In this file there are also tests for code style of all python scripts, using thepycodestylelibrary.Tests for the docs, samples, etc. can be found in tests/scripts/test_repo_files.py.
For running the tests locally, there is
tests/test.shwhich runs all above tests.Introduction to CDDL
In CDDL you define types from other types. Types can be defined from base types, or from other types you define. Types are declared with ‘
=‘, e.g.Foo = intwhich declares the typeFooto be an integer, analogous totypedef int Foo;in C. CDDL defines the following base types (this is not an exhaustive list):int: Positive or negative integeruint: Positive integerbstr: Byte stringtstr: Text stringbool: Booleannil: Nil/Null valuefloat: Floating point valueany: Any single elementCDDL allows creating aggregate types:
[]: List. Elements don’t need to have the same type.{}: Map. Key/value pairs as are declared as<key> => <value>or<key>: <value>. Note that:is also used for labels.(): Groups. Grouping with no enclosing type, which means that e.g.Foo = [(int, bstr)]is equivalent toFoo = [int, bstr]./: Unions. Analogous to unions in C. E.g.Foo = int/bstr/Barwhere Foo is either an int, a bstr, or Bar (some custom type).Literals can be used instead of the base type names:
Foo = 3, where Foo is a uint with the additional requirement that it must have the value 3.Foo = -100..100, where Foo is an int with value between -100 and 100.Foo = "hello", where Foo is a tstr with the requirement that it must be “hello”.Foo = false, where Foo is a bool which is always false.Base types can also be restricted in other ways:
.size: Works for integers and strings. E.g.Foo = uint .size 4where Foo is a uint exactly 4 bytes long..cbor/.cborseq: E.g.Foo = bstr .cbor Barwhere Foo is a bstr whose contents must be CBOR data decodable as the Bar type.An element can be repeated:
?: 0 or 1 time. E.g.Foo = [int, ?bstr], where Foo is a list with an int possibly followed by a bstr.*: 0 or more times. E.g.Foo = [*tstr], where Foo is a list containing 0 or more tstrs.+: 1 or more times. E.g.Foo = [+Bar].x*y: Between x and y times, inclusive. E.g.Foo = {4*8(int => bstr)}where Foo is a map with 4 to 8 key/value pairs where each key is an int and each value is a bstr.Note that in the zcbor script and its generated code, the number of entries supported via
*and+is affected by the default_max_qty value.Any element can be labeled with
:. The label is only for readability and does not impact the data structure in any way. E.g.Foo = [name: tstr, age: uint]is equivalent toFoo = [tstr, uint].See pet.cddl for CDDL example code.
Unsupported CDDL features
Not all features outlined in the CDDL specs RFC8610, RFC9090, and RFC9165 are supported by zcbor. The following is a list of limitations and missing features:
&()to turn groups into choices (unions).&()is supported when used with.bits.#x.y), except for tags (#6.y(foo)) which are supported.~).regexp,.ne,.default, and.withinfrom RFC8610..sdnv,.sdnvseq, and.oidfrom RFC9090..plus,.cat,.det,.abnf,.abnfb, and.featurefrom RFC9165.foo<a, b>).:for map keys.^or implicitly via:.Introduction to CBOR
CBOR’s format is described well on Wikipedia, but here’s a synopsis:
Encoded CBOR data elements look like this.
The available major types can be seen in
zcbor_major_type_t.For all major types, Values 0-23 are encoded directly in the Additional info, meaning that the Value field is 0 bytes long. If Additional info is 24, 25, 26, or 27, the Value field is 1, 2, 4, or 8 bytes long, respectively.
Major types
pint(0),nint(1),tag(6), andsimple(7) elements have no payload, only Value.pint: Interpret the Value as a positive integer.nint: Interpret the Value as a positive integer, then multiply by -1 and subtract 1.tag: The Value says something about the next non-tag element. See the CBOR tag documentation for details.simple: Different Additional info mean different things:falsesimple valuetruesimple valuenullsimple valueundefinedsimple valuelistormap.For
bstr(2),tstr(3),list(4), andmap(5), the Value describes the length of the Payload. Forbstrandtstr, the length is in bytes, forlist, the length is in number of elements, and formap, the length is in number of key/value element pairs.For
listandmap, sub elements are regular CBOR elements with their own Header, Value and Payload.lists andmaps can be recursively encoded. If alistormaphas Additional info 31, it is “indefinite-length”, which means it has an “unknown” number of elements. Instead, its end is marked by asimplewith Additional info 31 (byte value 0xFF).History
zcbor (then “cddl-gen”) was initially conceived as a code generation project. It was inspired by the need to securely decode the complex manifest data structures in the IETF SUIT specification. This is reflected in the fact that there are multiple zcbor tests that use the CDDL and examples from various revisions of that specification. Decoding/deserializing data securely requires doing some quite repetitive checks on each data element, to be sure that you are not decoding gibberish. This is where code generation could pull a lot of weight. Later it was discovered that the CBOR library that was designed to used by generated code could be useful by itself. The script was also expanded so it could directly manipulate CBOR data. Since CBOR, YAML, and JSON are all represented in roughly the same way internally in Python, it was easy to expand that data manipulation to support YAML and JSON.
Some places where zcbor is currently used:
Command line documentation
Added via
add_helptext.pyzcbor –help
zcbor code –help
zcbor validate –help
zcbor convert –help