Clemens Vasters

xRegistry hits RC3: one spec to describe — and generate — your entire messaging fabric

Wed, 15 Jul 2026 09:51:50 GMT

The CNCF xRegistry working group has shipped Release Candidate 3 (RC3) of the xRegistry specification set — the last feedback window before 1.0.
If you run anything event-driven, this is worth 5 minutes. xRegistry can now describe what you send to, and expect back from, practically any standard messaging channel — a Kafka topic, an AMQP queue, an MQTT topic tree — and generate the producer and consumer code from that description.
Why it mattersCloudEvents harmonized the shape of an event. It left four questions unanswered:
Discovery — Where do I produce or consume this? What endpoints exist?
Validation — How do I check an event's structure and its metadata?
Versioning — How do events and endpoints evolve without breaking consumers?
Serialization — What envelope and schema format is on the wire?
Schema registries answer some of this — but every broker ships its own, each vendor- and protocol-specific. When an event crosses brokers, clients wrestle with mismatched registries. xRegistry is the vendor-neutral layer that makes the metadata portable across all of them.
What it isA generic, extensible framework for managing metadata of any kind. At its core, an xRegistry is a simple tree:
Registry → Group → Resource → Version
Groups collect related Resources (think folders). Resources are the items of interest (think files), each optionally carrying a domain-specific document — like a schema — and optionally versioned. That's the whole model. Everything else is extension.
The defining trait: one symmetric shape, three forms. The same registry can live as:
A file — a single JSON manifest you check into Git next to your code.
A static website — one file per entity on plain storage (S3, GitHub Pages). No compute, no server logic.
A full API — a live service with search, filter, and write access for multiple teams.
You can start as a hand-written file and export your way up to an API as you grow — no rewrite, because the representations are designed to be symmetric.
What it does for messagingLayered on that core is a messaging and eventing catalog built from three companion specs — Endpoints, Messages, and Schemas.
Together they let you declare structured data flows over the protocols people actually use — AMQP, MQTT, Kafka, HTTP, and NATS — with or without the CloudEvents envelope, against a schema registry of your choice (JSON Schema, Avro, Protobuf, and more).
The payoff: a single manifest can fully describe an interface — the endpoints, the messages that flow over them, each message's Kafka key / AMQP subject / MQTT topic, a shared base CloudEvent definition, and the payload schemas each message points to.
From description to running codeThis is where it stops being documentation and starts saving work.
Generate data contracts — because schemas live beside the code in source control, tooling recreates typed data classes exactly in sync with the schema version.
Generate producers and consumers — from a vendor-agnostic definition, tooling emits broker-specific senders and receivers.
Validate on publish — a server can reject events that don't match the contract, killing poison messages at the door.
Shrink payloads — with schemas centralized, you can drop schema info from the wire and still analyze data accurately.
The reference codegen is an extensible, Jinja2-based framework that produces senders and receivers in several languages, honoring every declared attribute.
Overlay it on what you already havexRegistry doesn't demand you move your assets into it. Via the xrproxy project, an API server can project other registries into the xRegistry model and proxy them — leaving the source of truth in place.
In one live demo, a single xRegistry API sits in front of PyPI, npm, Maven, NuGet, Microsoft's OCI container registry, and the Model Context Protocol registry. Each has its own xRegistry model, so all six become one navigable, type-safe metadata graph — with cross-references between a container, the package inside it, the endpoint it exposes, and the event schemas that endpoint accepts.
See it runningFour things you can open right now:
The viewer + proxy — browse those six package/container registries through one consistent, type-safe API. 
The codegen gallery — messaging senders and receivers generated from xRegistry APIs and documents, across several languages: https://xregistry.io/codegen/gallery/
A live server — a hosted xrserver reference instance you can walk as a table or as raw JSON: https://xregistry.soaphub.org/ui/ (see CloudEvents for the messaging registry, APIs-Guru for the API-registry domain).
A real manifest — abstract Kafka/MQTT/AMQP endpoints bound to message groups, messages linking to JSON Structure schemas with equivalent Avro alongside. Zoom out to the parent folders and the MQTT, AMQP, and Kafka producer clients plus schema data classes are 100% generated from that manifest.
Go deeperA few design choices worth knowing before you build:
Federation by shadowing, not syncing. There's no synchronization API by design. Instead, registries layer: a local registry overrides or extends an underlying one without copying it — a file can shadow a folder, a folder can shadow an API, an API can shadow another API. For 1.0, treat shadowing as an emerging best practice — powerful, but still gaining real-world mileage.
Catalog, not container. A registry references far more than it stores. Documents can be embedded (inline or base64) or linked by URL — so you can catalog existing files where they already sit.
Flat groups, dotted names. The tree is deliberately one level deep (Groups don't nest). Hierarchy comes from dot-notation IDs like Contoso.ERP.Orders and from free-form labels for cross-cutting queries — without the complexity of nested containers.
Deliberately out of scope. Authentication and authorization are left to the hosting environment. So are formal relationships between channels — the focus for 1.0 is describing a single channel precisely first.
What's nextRC3 is the final call for feedback before xRegistry 1.0. If you work with events, schemas, or package/container/API registries, this is the moment to pressure-test the model against your use case.
Read the primer: https://xregistry.io/xreg/xregistryspecs/core-v1/docs/primer.html
Read the spec: https://github.com/xregistry/spec
The bottom linexRegistry turns scattered, broker-specific metadata into one portable graph you can discover, version, validate, and generate code from — as a document when at rest, an API when live, and the same shape either way. RC3 is your last chance to shape it before 1.0.

Avrotize & Structurize 3.0 is here!

Sat, 13 Dec 2025 12:31:27 GMT

Just before I vanish into the holiday break, I bundled up the release of Avrotize 3.0, a major update that introduces support for JSON Structure – a new intermediate schema format that bridges the gap between JSON Schema's familarity and expressiveness and Avro's type safety.

What's JSON Structure?

It's a JSON-based schema format with features Avro doesn't have:
- Discriminated unions with $discriminator
- Schema composition via $offers/$uses
- Validation keywords (minLength, pattern, etc.)
- Type inheritance with $extends

Why does this matter?

You can now convert the structures from OpenAPI specs, JSON Schema, or CDDL directly into production-ready entity classes for C#, Java, Python, Go, TypeScript, Rust, and C++ – without losing type fidelity along the way.

Introducing Structurize

If you're primarily working with JSON Structure and don't need Avro conversions, the structurize package is also available on PyPI. Currently an alias for avrotize, it will become a focused, standalone JSON Structure toolkit in a future release.

📦 pip install structurize

New in Avrotize 3.0:
- 35 new commands (now 62 total)
- OpenAPI 3.x → JSON Structure converter
- Database schema generation for SQL, Cassandra, Kusto, DynamoDB, Elasticsearch, CosmosDB, and more
- C# generators now support MessagePack, Protobuf-net, and CBOR
- Java/TypeScript test instance factories

See it in action

Check out the gallery with well over 100 conversion examples at

Home | Avrotize & Structurize

– see how schemas from various formats translate into generated code across all supported languages and many database schema dialects.

Key roadmap item for v4: Instance data transcoders; e.g. pipe XML data conforming to to a source XSD into the transcoder and get JSON output that conforms to the JSON Structure schema produced by the schema conversion. (That does largely already work with the code generators that support multiple encodings at the same time, with C# having most options)

📦 pip install avrotize
🔗clemensv/avrotize: Avrotize is a command-line tool for converting data structure definitions between different schema formats, using Apache Avro Schema as the integration schema model.

Happy holidays! 🎅

JSON Structure - An Introduction

Fri, 05 Dec 2025 09:13:19 GMT

This primer introduces JSON Structure, a modern, strictly typed data definition language that resembles JSON Schema, but has very different basic design principles.

JSON Structure focuses on defining data types with clear, deterministic constraints while supporting extensions for precise representations of numbers, dates, and more.

Common programming language concepts like reuse and composition are supported directly in the schema language, but while avoiding introduction of the entire complexity of object-oriented programming.

Conditional validation of JSON data against schemas with composition constructs like anyOf or allOf is availble in JSON Structure quite similar to JSON Schema, has been split out into optional extensions, allowing simple data definitions to remain lightweight and easy to understand.

1. Introduction

There are rapidly growing needs for a standardized (IETF RFC) schema language that can describe data types and structures and whose definitions map cleanly to programming language types and database constructs as well as into the popular JSON data encoding. The type model needs to reflect the needs of modern applications and it must allow rich annotations with semantic information that can be evaluated and understood by developers and by large language models (LLMs).

JSON Schema has been in development since ca. 2009 and has gone through several iterations. The industry has largely settled on “Draft 7” of JSON Schema, with subsequent releases seeing comparatively weak adoption. There’s substantial frustration that many developers have with JSON Schema because they try to use it for scenarios that it was not designed for. JSON Schema is a powerful document validation tool, but it is not a data definition language. Frustration with JSON Schema is widespread across the industry—and yet it remains popular due to lack of good alternatives.

Apache Avro is an alternative that has gained traction in the data community because it’s much easier to create tooling on top of. However, Avro’s type system is limited and its syntax is unfamiliar to most developers.

JSON Structure aims to combine the tooling-friendliness of Avro with the familiar type definition shapes of JSON Schema while providing an overall broader type system and data documentation/qualification capabilities that provide richer context for AI/LLMs trying to understand data structures.

While JSON Schema focuses on and excels at document validation, JSON Structure focuses on being a strong data definition language.

There are two major use-case scenarios for schema languages in general:

Validating JSON data against a schema to ensure conformity to a specific structure and constraints.
Declaring data types and structures in a machine-readable format that can be used to generate code, documentation, or other artifacts across platforms and languages.

JSON Schema provides powerful facilities for the first use-case. JSON Structure prioritizes the second use-case while still supporting the validation scenarios similar to JSON Schema through extensions.

With JSON Structure, we’ve designed a type system that addresses common challenges:

Clear mapping to programming language types
Support for more precise numeric types and date/time representations
Modular approach to extensions
Simplified cross-references between schema documents
Straightforward reuse patterns for types

This approach better supports code generation, database mappings, and integration with modern programming languages, while keeping the validation capabilities as optional extensions for those who need them.

2. Design Principles

JSON Structure is designed with the following principles in mind:

Scope: JSON Structure “Core” is meant to be a sufficient language for a “schema first” approach to defining application data structures in “polyglot” environments where such definitions are shared across application modules and tiers and databases and caches and other data stores. It is not a goal to be able to capture, say, all intricacies of a C++ class and express those. In that way, JSON Structure definitions are meant to be authoritative. Tools should be able to map from JSON Structure to programming language constructs and, within reasonable bounds, to database structures without losing structural information, but the reverse might not always be true.
Relation to JSON: JSON Structure is expressed in JSON and has built-in rules for how types are mapped to JSON encoding, but the data definitions can be seen as being independent of JSON. You can declare an object type in JSON Structure and use that type to code-generate a record class for an Object-Relational-Mapping framework and a corresponding database table without JSON encoding ever playing a role.
Familiarity: Millions of software developers recognize declarations of the shape “type”:“object”, “properties”: {…} as what a “schema” should look like and it’s often the only formal schema model they a have encountered. In spite of its shortcomings as a data definition language, JSON Schema’s most basic expressions for to declare a type are etched into people’s minds and any improvement that does not pick people up from where they are´will have it harder to gain substantial traction.
Simplicity: JSON Structure is designed to be simple and easy to understand. The schema language is designed to be easy to read and write, with a focus on clear and concise definitions. The type system is meant to easily align with concepts people use and need in their programming and database work, including safe numeric expressions and date and time expressions.
Pragmatism: JSON Structure is designed to be a pragmatic data definition language for application developers and not a language for metadata geeks. That means that when there is tension between providing type definitions for application use-cases and being able to describe or validate any imaginable JSON expression, the application use-cases win.
Optional Complexity: JSON Structure “Core” documents are always self-contained and simple to understand and process. If the user needs further capabilities like imports of definitions from external files, semantic annotations, or complex validation hints, they can opt into those by using companion specifications, potentially at the expense of compatibility with implementations of JSON Structure that are restricted to the “Core” specification.

3. Key Concepts

JSON Structure is designed to look and feel very much like the JSON Schema

drafts you already know, but some rules have been tightened up to make it easier to understand and use. Therefore, existing JSON Schema documents may need to be updated to conform to the new rules.

3.1. Differences to JSON Schema Drafts

Strict Typing: Every schema must explicitly specify its data type. For compound types (objects, arrays, sets), additional required metadata like a name and other required property definitions help enforce structured data.
Identifiers: Names of types and properties are restricted to the regular expression [A-Za-z_][A-Za-z0-9_]* to make them easier to map to code and database constructs. Map keys may additionally contain :, -, and . and may start with a digit.
Extended Types: In addition to JSON primitives such as string, number, boolean, and null, JSON Structure Core supports many extended primitive types (e.g., int32, int64, decimal, date, uuid) for high precision or format-specific data.
Compound Types: The compound types have been extended to include set, map, and tuple
- Object: Define structured data with a required name and a set of properties.
- Array: List of items where the items attribute references a declared type without inline compound definitions.
- Set: An unordered collection of unique elements.
- Map: A collection of key-value pairs where keys are strings and values are of a declared type.
- Tuple: A fixed-length array of elements where each element is of a declared type. It’s a more compact alternative to objects where the property names are replaced by the position in the tuple. Arrays or maps of tuples are especially useful for time-series data.
- Choice: A discriminated union of types, where one of the types is selected based on a discriminator property. The choice keyword is used to define the union and the selector property is used to specify the discriminator property.
Namespaces: Namespaces are a formal part of the schema language, allowing for more modular and deterministic schema definitions. Namespaces are used to scope type definitions.
Cross-Referencing: The $ref keyword has been limited to referencing named types that exist within the same document. It can no longer reference and insert arbitrary JSON nodes and it can no longer reference external documents. To reuse types from other documents, you now need to use the $import keyword from the optional import spec to import the types you need. Once imported, you can reference types with $ref.

3.2. Extensibility Model

JSON Structure is designed to be extensible. Any schema can be extended with custom keywords given that those do not conflict with the core schema language or companion specifications that are “activated” for the schema document. It’s recommended for custom keywords to have a compnay- or project-specific prefix to avoid such collisions. It’s not required to declare a formal meta-schema to use custom keywords, but it’s recommended to do so.

JSON Structure is designed to be modular. The core schema language is defined in the JSON Structure Core document. Companion specifications provide additional features and capabilities that can be used to extend the core schema language.

The abstract and $extends keywords enable controlled type extension, supporting basic object-oriented-programming-style inheritance while not permitting subtype polymorphism where a sub-type value can be assigned a base-typed property. This approach avoids validation complexities and mapping issues between JSON schemas, programming types, and databases.

There are two types of formal extensions: extensible types and add-in types.

3.3. Extensible Types

It’s fairly common that different types share a common basic set of properties. In JSON Schema itself, the description property is a good example of a property that is shared across all schema and non-schema objects.

An extensible type is declared as abstract and provides a set of common definitions to be shared by extensions. For example, a base type AddressBase MAY be extended by StreetAddress and PostOfficeBoxAddress via $extends. Because it’s abstract, AddressBase cannot be used directly anywhere as a type, however. Address extends AddressBase without adding any additional properties.

Example:

{
    "$schema": "https://json-structure.org/meta/core/v0/#",
    "definitions" : {
      "AddressBase": {
        "abstract": true,
        "type": "object",
        "properties": {
            "city": { "type": "string" },
            "state": { "type": "string" },
            "zip": { "type": "string" }
        }
      },
      "StreetAddress": {
        "type": "object",
        "$extends": "#/definitions/AddressBase",
        "properties": {
            "street": { "type": "string" }
        }
      },
      "PostOfficeBoxAddress": {
        "type": "object",
        "$extends": "#/definitions/AddressBase",
        "properties": {
            "poBox": { "type": "string" }
        }
      },
      "Address": {
        "type": "object",
        "$extends": "#/definitions/AddressBase"
      }
    }
}

3.4. Add-in Types

Add-in types allow modifying any existing type in a schema with additional properties or constraints. Add-ins are offered to the instance document author through the $offers keywords in the schema document. The instance document author can then enable the add-in by referencing the add-in name in the $uses keyword. When the instance document does so, the add-in properties are injected into the designated schema types before the schema is evaluated. Enabled add-ins are treated as if they were part of the schema from the beginning.

Add-ins can also be enabled in meta-schemas such that they are always applied to schemas that are based on the meta-schema.

A add-in type is declared as abstract and $extends a specific type that does not need to be abstract. For example, a add-in type DeliveryInstructions might be applied to any StreetAddress types in a document:

{
    "$schema": "https://json-structure.org/meta/core/v0/#",
    "$id": "https://schemas.vasters.com/Addresses",
    "$root": "#/definitions/StreetAddress",
    "$offers": {
        "DeliveryInstructions": "#/definitions/DeliveryInstructions"
    },
    "definitions" : {
      "StreetAddress": {
        "type": "object",
        "properties": {
            "street": { "type": "string" },
            "city": { "type": "string" },
            "state": { "type": "string" },
            "zip": { "type": "string" }
        }
      },
      "DeliveryInstructions": {
        "abstract": true,
        "type": "object",
        "$extends": "#/definitions/StreetAddress",
        "properties": {
            "instructions": { "type": "string" }
        }
      }
    }
}

Add-ins are applied to a schema by referencing the add-in name in the $uses keyword that is available only in instance documents. The $uses keyword is a set of add-in names that are applied to the schema for the document.

{
  "$schema": "https://schemas.vasters.com/Addresses",
  "$uses": ["DeliveryInstructions"],
  "street": "123 Main St",
  "city": "Anytown",
  "state": "QA",
  "zip": "00001",
  "instructions": "Leave at the back door"
}

3.5. Reusing Types across different Schema Documents

The prior versions of JSON Schema in effect allowed for $ref to reference arbitrary JSON nodes from the same or external document, which made references very difficult to understand and process, especially when the references were deep links into external schema documents and/or schema documents were cross-referenced using relative URIs to avoid committing to an absolute schema location.

JSON Structure has a more controlled approach to limit that complexity.

The $ref keyword can only reference named types that exist within the same document. It can no longer reference and insert arbitrary JSON nodes and it can no longer reference external documents.
To reuse types from other documents, you now need to use the $import keyword from the optional import spec to import the types you need into the scope of the schema document. Once imported, you use the imported types as if they were declared in the document. To avoid conflicts, you can import external types into a namespace.
Since types are imported and then referenced as if they were declared in the document, it’s also possible and permitted to “shadow” imported types with local definitions by explicitly declaring a type with the same name and namespace as as the imported type.

3.6. Core and Companion Specifications

The following documents are part of this JSON Structure proposal:

JSON Structure Core: Defines the core schema language for declaring data types and structures.
JSON Structure Alternate Names and Descriptions: Provides a mechanism for declaring alternate names and symbols for types and properties, including for the purposes of internationalization.
JSON Structure Symbols, Scientific Units, and Currencies: Defines keywords for specifying symbols, scientific units, and currency codes on types and properties.
JSON Structure Conditional Composition: Defines a set of conditional composition rules for evaluating schemas.
JSON Structure Validation: Specifies extensions to the core schema language for declaring validation rules.
JSON Structure Import: Defines a mechanism for importing external schemas and definitions into a schema document.

3.7. Meta-Schemas

Meta-schemas are JSON Structure documents that define the structure and constraints of schema documents themselves. Meta-schemas do not have special constructs beyond what is available in the core schema language.

A meta-schema is referenced in a schema document using the $schema keyword. The meta-schema defines the constraints that the schema document must adhere to.

The value of the $schema keyword is a URI that corresponds to the $id declared in the meta-schema document. The URI should be a resolvable URL that points to the meta-schema document, but for well-known meta-schemas, a schema processor will typically not actually fetch the meta-schema document. Instead, the schema processor will use the URI to identify the meta-schema and validate the schema document against its copy of the meta-schema.

A meta-schema can build on another meta-schema by $importing all of its definitions and then adding additional definitions or constraints or by shadowing definitions from the imported meta-schema.

The “core” meta-schema formally defines the elements described in the JSON Structure Core document. The “$id” of the core meta-schema is https://json-structure.org/meta/core/v0.

The “extended” meta-schema extends the core meta-schema with all additional features and capabilities provided by the companion specifications and offers those features to schema authors. The “$id” of the extended meta-schema is https://json-structure.org/meta/extended/v0.

The “validation” meta-schema enables all add-ins defined in the extended meta-schema.

4. Using Structure Core

This section introduces JSON Structure by example.

4.1. Example: Declaring a simple object type

Here is an example of a simple object type definition:

{
    "$schema": "https://json-structure.org/meta/core/v0/#",
    "type": "object",
    "name": "Person",
    "properties": {
        "firstName": { "type": "string" },
        "lastName": { "type": "string" },
        "dateOfBirth": { "type": "date" }
    },
    "required": ["firstName", "lastName"]
}

If you are familiar with JSON Schema, you will instantly recognize the structure of this schema. The type attribute specifies that this is an object type. The properties attribute lists the properties of the object, and the required attribute lists the properties that are required.

There are a few differences from prior version of JSON Schema. The name attribute is new and is required for the root type. The type attribute is also required and no longer implied to be object if not present. You may also notice that the “dateOfBirth” property uses the new date type, which is an extended native type.

4.2. Example: Declaring Primitive and Extended Types

Below is an example schema that defines a simple profile with a few more extended types:

{
    "$schema": "https://json-structure.org/meta/core/v0/#",
    "type": "object",
    "name": "UserProfile",
    "properties": {
        "username": { "type": "string" },
        "dateOfBirth": { "type": "date" },
        "lastSeen": { "type": "datetime" },
        "score": { "type": "int64" },
        "balance": { "type": "decimal", "precision": 20, "scale": 2 },
        "isActive": { "type": "boolean" }
    },
    "required": ["username", "birthdate"]
}

The int64 type is an extended type that represents a 64-bit signed integer. The decimal type is another extended type that represents a decimal number with a specified precision and scale. The datetime type is an extended type that represents a date and time value.

5. Example: Declaring inline compound types

This is an example of a type that is declared inline. This is useful for compound types that are not reused elsewhere in the schema. The address property of the UserProfile type references the inline Address type. This type cannot be referenced from other types in the schema.

{
    "$schema": "https://json-structure.org/meta/core/v0/#",
    "type": "object",
    "name": "UserProfile",
    "properties": {
        "username": { "type": "string" },
        "dateOfBirth": { "type": "date" },
        "lastSeen": { "type": "datetime" },
        "score": { "type": "int64" },
        "balance": { "type": "decimal", "precision": 20, "scale": 2 },
        "isActive": { "type": "boolean" },
        "address": {
            "type": "object",
            "name": "Address",
            "properties": {
                "street": { "type": "string" },
                "city": { "type": "string" },
                "state": { "type": "string" },
                "zip": { "type": "string" }
            },
            "required": ["street", "city", "state", "zip"]
        }
    },
    "required": ["username", "birthdate"]
}

5.1. Example: Declaring reusable types in `definitions`

To define reusable types, you can use the definitions keyword to define types that can be referenced by other types in the same document. Here is an example:

{
    "$schema": "https://json-structure.org/meta/core/v0/#",
    "type": "object",
    "name": "UserProfile",
    "properties": {
        "username": { "type": "string" },
        "dateOfBirth": { "type": "date" },
        "lastSeen": { "type": "datetime" },
        "score": { "type": "int64" },
        "balance": { "type": "decimal", "precision": 20, "scale": 2 },
        "isActive": { "type": "boolean" },
        "address": { "type" : { "$ref": "#/definitions/Address" } }
    },
    "required": ["username", "birthdate"],
    "definitions": {
        "Address": {
            "type": "object",
            "properties": {
                "street": { "type": "string" },
                "city": { "type": "string" },
                "state": { "type": "string" },
                "zip": { "type": "string" }
            },
            "required": ["street", "city", "state", "zip"]
        }
    }
}

In this example, the Address type is declared in the definitions section and can be referenced by other types in the same document using the $ref keyword. Mind that the $ref keyword can now only reference types declared in the definitions section of the same document. The keyword can only be used where a type is expected.

5.2. Example: Structuring types with namespaces

Namespaces are used to scope type definitions. Here is an example of how to use namespaces to structure your types, with two differing Address types:

{
    "$schema": "https://json-structure.org/meta/core/v0/#",
    "type": "object",
    "name": "UserProfile",
    "properties": {
        "username": { "type": "string" },
        "dateOfBirth": { "type": "date" },
        "networkAddress": { "type" : { "$ref": "#/definitions/Network/Address" } },
        "physicalAddress": { "type": { "$ref": "#/definitions/Physical/Address" } }
    },
    "required": ["username", "birthdate"],
    "definitions": {
        "Network": {
            "Address": {
                "type": "object",
                "properties": {
                    "ipv4": { "type": "string" },
                    "ipv6": { "type": "string" }
                }
            }
        },
        "Physical": {
            "Address": {
                "type": "object",
                "properties": {
                    "street": { "type": "string" },
                    "city": { "type": "string" },
                    "state": { "type": "string" },
                    "zip": { "type": "string" }
                },
                "required": ["street", "city", "state", "zip"]
            }
        }
    }
}

5.3. Example: Using an Array Type

This example shows how to declare an array of strings, which is not much different from defining an object:

{
    "$schema": "https://json-structure.org/meta/core/v0/#",
    "type": "array",
    "items": { "type": "string" }
}

You can also declare an array of a locally declared compound type, but you can not reference the type from elsewhere in the schema:

{
    "$schema": "https://json-structure.org/meta/core/v0/#",
    "type": "array",
    "items": {
        "type": "object",
        "properties": {
            "firstName": { "type": "string" },
            "lastName": { "type": "string" },
            "dateOfBirth": { "type": "date" }
        },
        "required": ["firstName", "lastName"]
    }
}

To declare an array of a reusable type, you can use the $ref keyword:

{
    "$schema": "https://json-structure.org/meta/core/v0/#",
    "type": "array",
    "items": { "type" : { "$ref": "#/definitions/Person" } },
    "definitions": {
        "Person": {
            "type": "object",
            "name": "Person",
            "properties": {
                "firstName": { "type": "string" },
                "lastName": { "type": "string" },
                "dateOfBirth": { "type": "date" }
            },
            "required": ["firstName", "lastName"]
        }
    }
}

5.4. Example: Declaring Maps

This example shows how to declare a map of strings to Color objects:

{
    "$schema": "https://json-structure.org/meta/core/v0/#",
    "type": "map",
    "values": { "type": { "$ref": "#/definitions/Color" } },
    "definitions": {
        "Color": {
            "type": "object",
            "name": "Color",
            "properties": {
                "red": { "type": "int32" },
                "green": { "type": "int32" },
                "blue": { "type": "int32" }
            },
            "required": ["red", "green", "blue"]
        }
    }
}

Instance data for this schema might look like this:

{
    "rose": { "red": 255, "green": 0, "blue": 0 },
    "sky": { "red": 0, "green": 191, "blue": 255 },
    "grass": { "red": 0, "green": 128, "blue": 0 },
    "sun": { "red": 255, "green": 215, "blue": 0 },
    "cloud": { "red": 255, "green": 255, "blue": 255 },
    "moon": { "red": 192, "green": 192, "blue": 192 }
}

5.5. Example: Declaring Sets

This example shows how to declare a set of strings:

{
    "$schema": "https://json-structure.org/meta/core/v0/#",
    "type": "set",
    "items": { "type": "string" }
}

Sets differ from arrays in that they are unordered and contain only unique elements. The schema above would match the following instance data:

["apple", "banana", "cherry"]

5.6. Example: Declaring Tuples

Tuples are fixed-length arrays with named elements, declared via the tuple keyword.

{
    "$schema": "https://json-structure.org/meta/core/v0/#",
    "type": "tuple",
    "name": "PersonTuple",
    "properties": {
        "firstName": { "type": "string" },
        "age":       { "type": "int32" }
    },
    "tuple": ["firstName", "age"]
}

An instance of this tuple type:

["Alice", 30]

5.7. Example: Declaring Choice Types

Choice types define discriminated unions via the choice keyword. Two forms are supported:

5.8. Tagged Unions

Tagged unions represent the selected type as a single-property object:

{
    "$schema": "https://json-structure.org/meta/core/v0/#",
    "type": "choice",
    "name": "StringOrNumber",
    "choices": {
        "string": { "type": "string" },
        "int32":  { "type": "int32" }
    }
}

Valid instances:

{ "string": "Hello" }
{ "int32":  42 }

5.9. Inline Unions

Inline unions extend a common abstract base type and use a selector property:

{
    "$schema": "https://json-structure.org/meta/core/v0/#",
    "type": "choice",
    "name": "AddressChoice",
    "$extends": "#/definitions/Address",
    "selector": "addressType",
    "choices": {
        "StreetAddress":     { "type": { "$ref": "#/definitions/StreetAddress" } },
        "PostOfficeBoxAddress": { "type": { "$ref": "#/definitions/PostOfficeBoxAddress" } }
    },
    "definitions": {
        "Address": {
            "abstract": true,
            "type": "object",
            "properties": {
                "city": { "type": "string" },
                "state":{ "type": "string" },
                "zip":  { "type": "string" }
            }
        },
        "StreetAddress": {
            "type": "object",
            "$extends": "#/definitions/Address",
            "properties": { "street": { "type": "string" } }
        },
        "PostOfficeBoxAddress": {
            "type": "object",
            "$extends": "#/definitions/Address",
            "properties": { "poBox": { "type": "string" } }
        }
    }
}

Instance of this inline union:

{
    "addressType": "StreetAddress",
    "street":      "123 Main St",
    "city":        "AnyCity",
    "state":       "AS",
    "zip":         "11111"
}

6. Using Companion Specifications

The JSON Structure Core specification is designed to be extensible through companion specifications that provide additional features and capabilities.

The extended schema that includes all companion specifications is identified by the https://json-structure.org/meta/extended/v0 URI. Each companion specification is identified by a unique identifier that can be used in the $uses attribute to activate the companion specification for the schema document.

The feature identifiers for the companion specifications are:

JSONStructureAlternateNames: Alternate names and descriptions for properties and types.
JSONStructureUnits: Symbols, scientific units, and currencies for numeric properties.
JSONStructureImports: Importing types from other schema documents.
JSONStructureValidation: Validation rules for JSON data.
JSONStructureConditionalComposition: Conditional composition and validation rules.

6.1. Example: Using the `altnames` Keyword

The JSON Structure Alternate Names and Descriptions companion specification introduces the altnames keyword to provide alternate names for properties and types. Alternate names provide additional mappings that schema processors may use for encoding, decoding, or user interface display.

Here is an example of how to use the altnames keyword:

{
    "$schema": "https://json-structure.org/meta/extended/v0/#",
    "$uses": ["JSONStructureAlternateNames"],
    "Person": {
        "type": "object",
        "altnames": {
            "json": "person_data",
            "lang:en": "Person",
            "lang:de": "Person"
        },
        "properties": {
            "firstName": {
                "type": "string",
                "altnames": {
                    "json": "first_name",
                    "lang:en": "First Name",
                    "lang:de": "Vorname"
                }
            },
            "lastName": {
                "type": "string",
                "altnames": {
                    "json": "last_name",
                    "lang:en": "Last Name",
                    "lang:de": "Nachname"
                }
            }
        },
        "required": ["firstName", "lastName"]
    }
}

Each named type or property in the schema has been given an altnames attribute that provides alternate names for the type or property.

The json key is used to specify alternate names for JSON encoding, meaning that if the schema is used to encode or decode JSON data, the alternate key MUST be used instead of the name in the schema.

Keys beginning with lang: are reserved for providing localized alternate names that can be used for user interface display. Additional keys can be used for custom purposes, subject to no conflicts with reserved keys or prefixes.

6.2. Example: Using the `altenums` Keyword

The JSON Structure Alternate Enumerations companion specification introduces the altenums keyword to provide alternative representations for enumeration values defined by a type using the enum keyword. Alternate symbols allow schema authors to map internal enum values to external codes or localized display symbols.

Here is an example of how to use the altenums keyword:

{
    "$schema": "https://json-structure.org/meta/extended/v0/#",
    "$uses": ["JSONStructureAlternateNames"],
    "type": "object",
    "name": "Color",
    "properties": {
        "name": { "type": "string" },
        "value": { 
            "type": "string", 
            "enum": ["red", "green", "blue"],
            "altenums": {
                "lang:en": {
                    "red": "Red",
                    "green": "Green",
                    "blue": "Blue"
                },
                "lang:de": {
                    "red": "Rot",
                    "green": "Grün",
                    "blue": "Blau"
                }
            }
        }
    }
}

In this example, the value property has an enum attribute that defines the possible values for the property. The altenums attribute provides alternative names for each enumeration value. The lang:en key provides English names for the enumeration values, and the lang:de key provides German names.

6.3. Example: Using the `unit` Keyword

The JSON Structure Symbols, Scientific Units, and Currencies companion specification introduces the unit keyword to provide a standard way to specify the unit of measurement for numeric properties. The unit keyword allows schema authors to specify the unit of measurement for numeric properties and provides a standard way to encode and decode numeric values with units.

Here is an example of how to use the unit keyword:

{
    "$schema": "https://json-structure.org/meta/extended/v0/#",
    "$uses": ["JSONStructureUnits"],
    "type": "object",
    "name": "Pressure",
    "properties": {
        "value": { "type": "number", "unit": "Pa" }
    }
}

In this example, the value property has a unit attribute that specifies the unit of measurement for the property. The unit of measurement is specified as a string value. In this case, the unit of measurement is “Pa” for Pascals.

6.4. Example: Using the `currency` Keyword

The JSON Structure Symbols, Scientific Units, and Currencies companion specification also introduces the currency keyword to provide a standard way to specify the currency for monetary properties. The currency keyword allows schema authors to specify the currency for monetary properties and provides a standard way to encode and decode monetary values with currencies.

Here is an example of how to use the currency keyword:

{
    "$schema": "https://json-structure.org/meta/extended/v0/#",
    "$uses": ["JSONStructureUnits"],
    "type": "object",
    "name": "Price",
    "properties": {
        "value": { "type": "decimal", "precision": 20, "scale": 2, "currency": "USD" }
    }
}

In this example, the value property has a currency attribute that specifies the currency for the property. The currency is specified as a string value. In this case, the currency is “USD” for US Dollars.

7. Using Validation

The companion specifications for conditional composition and validation provide additional constructs for defining conditional validation rules and composing that resemble those found in prior versions of JSON Schema. However, those have been split out into optional extensions to keep the core schema language simple.

7.1. Example: Using Conditional Composition

The JSON Structure Conditionals companion specification introduces conditional composition constructs for combining multiple schema definitions. In particular, this specification defines the semantics, syntax, and constraints for the keywords allOf, anyOf, oneOf, and not, as well as the if/then/else conditional construct.

The specification has several examples that show how to use the conditional composition keywords.

7.2. Example: Using Validation Rules

The JSON Structure Validation companion specification introduces additional validation rules for JSON data.

8. SDK and Tooling Ecosystem

A schema language without tooling remains theoretical. For developers to adopt JSON Structure, they need to be able to tell whether schemas are valid, whether documents conform to a given schema, and they need to be able to turn schemas into data structures they can code against.

The JSON Structure SDKs provide native implementations for schema validation (is this schema well-formed?) and instance validation (does this JSON document conform to this schema?) across all major programming languages:

Language	Package	Install
Python	json-structure	`pip install json-structure`
.NET/C#	JsonStructure	`dotnet add package JsonStructure`
Java	json-structure	Maven Central
TypeScript/JS	@aspect/json-structure	`npm install @aspect/json-structure`
Go	github.com/json-structure/sdk/go	`go get github.com/json-structure/sdk/go`
Rust	json-structure	`cargo add json-structure`
C/C++ (C99)	Build from repo	CMake

For languages with runtime introspection (Java, .NET/C#, Python), the SDKs also include schema exporters that produce JSON Structure schemas from existing runtime data structures—useful for adopting JSON Structure incrementally in existing codebases.

The SDK repository is at https://github.com/json-structure/sdk

9. Visual Studio Code Extension

The JSON Structure extension for Visual Studio Code, built on the TypeScript SDK, provides a first-class editing experience for .struct.json schema files:

Schema validation: Real-time feedback on schema syntax and structure
Instance validation: Validate JSON documents against JSON Structure schemas
IntelliSense: Auto-completion for keywords, types, and references
Hover documentation: Contextual help for all schema elements

Install from the VS Code Marketplace or search for “JSON Structure” in the Extensions view.

10. Structurize: Schema Conversion and Code Generation

The Structurize tool (a persona of the Avrotize tool) enables robust, predictable conversion between numerous schema formats and generates code for multiple target languages. JSON Structure serves as a second “pivot” schema model for this tool, meaning you can convert from many formats to JSON Structure, and from JSON Structure to many targets—preserving the full richness of the type model.

Installation

pip install structurize
# or
pip install avrotize

For the moment, the tools have an identical set of commands, but they may eventually be split. There will eventually be a bundled installer (as for the Azure CLI) so that you don’t need Python preinstalled.

Example Conversions

From	To	Command
JSON Structure	C#	`structurize jsonstruct2cs schema.struct.json --out Models.cs`
JSON Structure	Python	`structurize jsonstruct2py schema.struct.json --out models.py`
JSON Structure	Java	`structurize jsonstruct2java schema.struct.json --out-dir ./src`
JSON Structure	Go	`structurize jsonstruct2go schema.struct.json --out models.go`
JSON Structure	C++	`structurize jsonstruct2cpp schema.struct.json --out models.hpp`
JSON Structure	Protobuf	`structurize jsonstruct2proto schema.struct.json --out schema.proto`
JSON Structure	PostgreSQL	`structurize jsonstruct2pgsql schema.struct.json --out schema.sql`

There are many more database dialects and target formats available. The full docs are in the Avrotize repo README.

Chaining Conversions

The tool is built such that you can pipe conversions, so you could go from ASN.1 via Avro to JSON Structure to C# with minimal loss of type info:

# ASN.1 → Avro → JSON Structure → C#
structurize asn2avro schema.asn | structurize avro2jsonstruct | structurize jsonstruct2cs --out Models.cs

11. Resources and Next Steps

Specifications (IETF Internet Drafts)

All JSON Structure specifications are published as IETF Internet Drafts:

JSON Structure Core - The core schema language
JSON Structure Import - Importing types from other documents
JSON Structure Alternate Names - Alternate names and i18n
JSON Structure Units - Symbols, scientific units, currencies
JSON Structure Validation - Validation constraints
JSON Structure Conditional Composition - allOf, anyOf, oneOf, if/then/else

All drafts are available at: https://datatracker.ietf.org/doc/search?name=draft-vasters-json-structure

SDK and Tools

Resource	Link
SDK Repository	https://github.com/json-structure/sdk
VS Code Extension	Search “JSON Structure” in VS Code Marketplace
Structurize/Avrotize	`pip install structurize` or https://github.com/clemensv/avrotize

Sample Schemas

The samples folder in this repository contains working examples organized by specification:

samples/core/ - Core specification examples with validation scripts
samples/import/ - Import specification examples

Getting Started

Install the VS Code extension for editing support with validation and IntelliSense
Install the SDK for your language to validate schemas and instances programmatically
Try the samples to understand the schema patterns
Use Structurize to convert existing schemas or generate code for your target language

Agent Autonomy

Mon, 18 Aug 2025 12:10:16 GMT

Autonomy is what makes AI agents reliable building blocks.

It means an agent is a self-contained unit of management, deployment, and ownership that can operate independently while participating in a larger system. It controls its own state, evolves on its own schedule, and interacts with others only through well-defined contracts. It has its own lifecycle, its own operational envelope, and its own accountability boundaries.

Sounds familiar? The principles of service orientation are alive and well and proven and will also apply to collaborative agents.

Autonomy means:
- Owns its state and logic → No shared DB tables, no shared caches, no hidden in-memory state tied to another agent’s runtime. If you want something, you ask through its defined interface.
- Runs on its own lifecycle and scaling plan → Deploy, update, restart, or scale it without coordinating with others.
- Exposes clear RPC or messaging interfaces → All interactions happen through an explicit, documented contract.
- Honors its contract → Can be evolved internally without breaking external expectations.

When agents have these properties, we can assemble collaborative processes the way we plug components into a system bus — mixing humans, LLM-powered reasoning agents, and traditional services without tight coupling.

Autonomy isn’t just an architectural preference — it’s the foundation for resilience, scalability, and adaptability.

It means:
- Agents can be developed by different teams.
- Deployed on different schedules.
- Governed by different owners.
- Still work together without friction.
- Failures stay contained.
- Upgrades don’t ripple across the network.
- New capabilities can be added without destabilizing what’s already running.

Without autonomy, your “agent network” is just a fragile, distributed monolith — every change is a negotiation, scaling breaks under load, and one bad actor can take the whole thing down.

And it is essential for autonomous agents to also be temporally decoupled from others and to not immediately tie the reliability/availability of an agent to that of another. You achieve that by preferring asynchronous communication paths wherever feasible. Surround agents with moats of queues and embrace event-driven architecture for creating process observability.

An Interface Design Test for Collaborative Agents

Mon, 18 Aug 2025 08:21:30 GMT

An "agent" is a participant in a collaboration that autonomously solves a given aspect of a problem in the context of a system. An agent can be implement by a deterministic software module, by a person (human-in-the-loop), or by an AI component (AI-in-the-loop). Some implementations may employ multiple of these at once or as fallback options.

I propose the following design test for the interfaces of collaborating agents, inspired by the Turing Test:

If you communicate through an I/O interface with an agent, and you cannot distinguish by the form of the I/O interface or its data flow shape or behavior or even its temporal behavior whether the agent is implemented by deterministic code, by AI-in-the-loop, by Human-in-the-loop, or by a federation of collaborating agents, then your interface design is correct.

The motivation of this test is that you will generally understand the problem you are trying to solve and you will understand what inputs you can provide and what outputs you expect, but you may not know what implementation choice ends up being the best for a problem once you consider cost and performance and accuracy and user satisfaction over time. Problems might not yet be solvable using AI but they will be in the future, or you might think that AI is the best choice now, but you may have to fall back to other options for some reason.

One of the consequences from this test is that the interface must flow structured data and not an unstructured "natural language interface", which would require an LLM or a similar processor to break down the input for a deterministic implementation. Each agent must have a clear and well-defined input and output interface. Natural language input must be broken down to match that interface.

CNCF xRegistry 1.0 RC1 - What. Why. What might be next.

Wed, 16 Apr 2025 12:49:29 GMT

Two weeks ago, during a very busy conference week that saw CNCF KubeCon, Hannover Messe, and FabCon all take place simultaneously, with me being in Hannover, the CNCF Serverless Working Group that I am part of, released the first release candidate of xRegistry. This is a significant milestone for the project, and I wanted to take a moment to share what xRegistry is, why it matters, and what’s next for the project.
OriginsThe xRegistry project is a sibling project of CNCF CloudEvents, produced by the same working group and, for the most part, the same people.
After the CloudEvents 1.0 release in 2019, the working group vowed to avoid breaking changes and give CloudEvents time and stability for adoption. Since then, we've relased 1.0.1, 1.0.2 as "errata" releases with 1.0.3 being the current "work in progress" version for further corrections. None of those releases made any substantial changes to the core spec, but we've made some additions and a few tweaks in the attribute extensions and transport binding and event format areas.
ObservationsAs folks were starting to implement CloudEvents in serious projects, we made a few observations:
First, the data section of a CloudEvent generally contains structured data that is serialized in a specific encoding (JSON, XML, Avro, Protobuf, etc.). Some of these encodings require schema documents to be able to even understand the encoded data, as it is the case for Avro and Protobuf.
For the CloudEvents envelope, we had a formal binding and schema for Avro in the v1.0 release and added a Protobuf binding in the v1.0.1 release.
Since our goal for CloudEvents is to promote interoperability and vendor neutrality, the question quickly arose on how to enable applications to share schema documents for the data section of a CloudEvent if developers wanted to use one of these efficient and binary but schema-dependent encodings. Some schema registry products/projects were around at the time, but there were no vendor-neutral specifications or standards for them.
Second, we observed that when applications were raising CloudEvents, the way how those CloudEvents were made known to developers was generally through prose and examples in documentation. There was no formal, machine readable way to declare and document the shape of a CloudEvent: What is the type value, what does the source value look like, what are the required and optional standards and extension attributes for this specific event type, and what is the schema for the data section?
Third, we heard that it would be very useful for developer tools to be able to know about event sources and sinks that exist in their ecosystem, so that applications can be built that can react to events from those sources and send events to those sinks.
ExplorationsAs a result, we started some exploratory work in the working group on "schema registry" and "discovery". Mike Helmick from LinkedIn submitted a "Discovery API" in March 2020. I proposed a "Schema registry API" in May 2020.
After the chaos of the pandemic, we starts a sub-working group to make progress in that area and to explore the three aspects I mentioned above: How do I share schema? How do I catalog and declare events? How do I discover event sinks and sources?
As we made progress on this, we realized that these three sub-projects were so interdependent that it would be beneficial if these three registries could be, optionally, collocated behind a single API facade. We also realized that an API alone would not be sufficient, but that we would also want for the same metadata to be available in document format such that it could be exchanged or used to drive tools like code generators.
As we then worked on aligning the APIs, we found that we had a lot of commonalities across the three registries, and that we could actually build a single registry API that would be able to serve all three use cases, whereby the differences could be expressed in a lightweight configuration-model that the registry server could use to adapt the API to the specific use case.
Once we arrived at this conclusion, we also realized that we could use the same configuration model for any other metadata registry use case. Thus, the extensible registry, xRegistry, came into being.
xRegistryThe core specification of xRegistry describes a document format and a REST API that can be used to manage and interact with metadata registries. The core specification also defines the configuration model that allows to define the registry behavior and the specific sub-registry metadata content. If the hasdocument model attribute is true for a resource type, the registry acts as a document store with accompanying metadata. If the value is false, it's a pure structured data store.
A key principle across xRegistry is that all resources are managed inside of groups. A group is a logical collection of resources that can be used for organization and/or access control. It's possible to cross-reference resources between groups, such that one resource can logically occur in multiple groups.
The schema specification defines a sub-registry for serialization and validation schema documents. It enables the versioning model of xRegistry such that all versions of a schema document are kept in the registry and can be referenced by their version. The registry has a notion of a "default" schema version, which is the one that is returned when no version is specified. The registry is also able to maintain version lineage information and provides a compatibility policy mechanism that implementations can enforce.
The message definitions specification defines a sub-registry for event and message definitions. Initially, we only allowed for declaring CloudEvents, but we quickly realized that declaring metadata for all other major messaging protocols (AMQP, MQTT, Kafka, etc.) would be useful as well, even in combination with CloudEvents. The message definition registry allows declaring both protocol-specific metadata and envelope metadata for events, including field-level descriptions and type information. String-typed metadata fields can be declared as being URI templates, which allows declaring placeholders and simple filter wildcards. Each message definition can be linked to a schema from the schema registry or can embed a local schema for the payload.
The grouping mechanism of xRegistry requires that message definitions are grouped, which is a very convenient way to organize message definitions that apply to a specific communication channel. A message definition group can therefore form a logical interface for a specific communication channel, e.g. a Kafka topic or an AMQP queue.
The endpoint registry specification defines a sub-registry for event sources (consumer endpoints) and sinks (publisher endpoints) as well as for "subscriber endpoints" that advertise protocol-specific endpoints to establish a consumer relationship, e.g. the address of a Kafka broker to create a new consumer group. The specification explicitly defines protocol-specific metadata for various protocols to help with interoperability. Each endpoint can reference one or more message definition groups from the message definition registry. Each endpoint also forms it own message definition group, meaning it can embed one or more message definitions that are specific to the endpoint.
There are some examples of the document form of xRegistry in the samples section of the project repository. When you look through those, you will likely find that while the specification documents are very detailed, concrete document examples are quite simple to read. The schemastore.org example shows an xRegistry overlay for the entire schema collection of that site.
There is chatter about xRegistry competing with AsyncAPI. I'd say the efforts have very different goals. The xRegistry message definition registry's "message groups" yield a similar concept to AsyncAPI channels, but xRegistry does not define a model for how message flows and channels relate. xRegistry is a metadata catalog first and foremost; its ability to define contracts for channels is a byproduct of its structure.
How is this useful to me? Right now?Follow the links and study the following three examples of xRegistry documents:
Camera CloudEvents/JSON Schema
Wind Generator Kafka/Avro
Waiter Boiler MQTT5/JSON 
What you'll find is that these documents are formal definitions of what flows over a Kafka topic, a set of MQTT topics and a generic CloudEvents channel. If you are using Kafka or MQTT or AMQP or NATS or HTTP WebHooks or CloudEvents today, xRegistry gives you an instantly useful way to formally document what's going on in that Kafka topic and in that AMQP queue. Which headers or which key identify which kind of message and what schema is associated with it. 
In a way, this is like WSDL (there's an old term!) or OpenAPI for asynchronous eventing/messaging channels, but for the most popular protocol messages and envelopes and schemas and encodings all at once.  Instantly useful as a documentation tool.
A complete reference implementation of the xRegistry API is part of the project, which means that you can also stand up a server already to manage your metadata. Front this server with a proxy or API gateway that handles authorization and you've got a very flexible metadata solution.
Down the road, and that ought not to shock you knowing that I work at Microsoft, these formal declarations of what flows on a messaging/eventing channel will become very concretely useful, because you'll be able to tell Azure and Fabric about your channels with them and our tooling will be able to help you a lot better with wrangling your fast moving data.
What's Next?Now, and this leads me to the "what's next" part, I do think there's a need for becoming formal about relationships between channels and for defining message flows. I do not think that a request/response and client/server "API" notion is the right starting point and approach for that.
We are looking at a future of collaborating agentic applications that are strung together through asynchronous communication channels. An agent is getting a job to handle and enlists a number of other agents to help with that job. The work on that job may take a long time and results may be produced progressively over multiple responses or with intermediate states change reports.
The ISO 20022 set of standards for financial services defines a message flow diagram for each described business process and associated set of message definitions. These message flows are not APIs, they are often fairly complex, multi-party document exchanges.
The ITU Z.120 "Message Sequence Chart" standard is fairly ancient and a long and arduous read, but it is good prior art to look at for a formalism to describe message flows. I believe that "sequence diagrams as a contract" is a missing piece in the puzzle of strongly-typed, interoperable, and formally defined asynchronous communication.
xRegistry's endpoint definitions with associated message definition groups and schemas are well suited to declare the "instance contracts" in the message sequence chart shown below, but we need an extra formal layer to describe the message flow itself.

From ITU Z.120
Whatever that contract document will look like, it's metadata and therefore will be managable inside of xRegistry.

A thought experiment: "New JSON Schema"

Wed, 19 Feb 2025 11:59:10 GMT

JSON Schema has been in development since ca. 2009 and has gone through several iterations. Yet, there is still no IETF RFC anyone could really lean on as a standard. Practitioners are largely using "Draft 7" of JSON Schema and the subsequent releases have seen comparatively little adoption.

The quality of the specs in terms of clarity and precision has been a major issue and the JSON Schema project has stood up a website that explains the spec in more detail than the spec itself.

Structured metadata is becoming rapidly more important in the world of APIs and LLMs. Large language models can operate better in structured data if they are fed with rich context information and schema documents can provide such context.

JSON Schema should play a big role in this context, but its complexity and ambiguity and the lack of a finalized standard are major obstacles. Efforts like OpenAPI lean on subsets of JSON Schema and had to invent their own extensions to cover gaps. There are no two implementations of JSON Schema code generators that agree on the output mapping of conditional JSON Schema structures to code, including those for OpenAPI.

Worse yet, all of these tools (need to) give up on complex JSON Schema constructs at some point and that point is different for each implementation. It's close to impossible to write JSON Schemas that can be used reliably for code generation in a cross-platform and cross-language way unless you scope out a substantial portion of the JSON Schema language.

There are two major use-case scenarios for JSON Schema and schema languages in general:

Users want to validate JSON data against a schema to ensure that the data conforms to a specific structure and set of constraints.
Users want to declare data types and structures in a machine-readable format that can be used to generate code, documentation, or other artifacts in a cross-platform, cross-language way.

JSON Schema has enormously powerful facilities for the first use-case right in its core. All that power comes at the expense of the second use-case.

The existing drafts of JSON Schema define a pattern-matching language for schema processors that is applied to JSON data as it is being validated. It is not a data definition language. It is a validation language that embeds elements that only look like data definition capabilities. An object declaration in JSON Schema is a matching expression for a JSON object that contains the properties defined in the schema; it does not define an object type.

Conditional composition constructs like allOf, anyOf, oneOf, not, and if/then/else are defined in the core schema language. As powerful as they are, conditional composition of data types is generally not a thing in databases or programming languages, which means that any use of these constructs makes mapping from and to code and databases hard and in many cases impossible while preserving the schema semantics.

JSON Schema allows for $ref to reference arbitrary JSON nodes (any of which are schemas) from the same or external document, which adds to the complexity. A single JSON Schema might have dozens of external links to content, strewn across the document, making it very difficult to understand as well as hard and potentially unsafe to process.

There are also confusing conflicts and overlaps. For instance, JSON Schema has a concept of a type union that can only be used for primitive types. Users frequently side-step that limitation through a oneOf construct that behaves equivalent to a type union for validation and thus there are factually two type union constructs.

Enumerations are first-class constructs in many programming languages and generally map symbols to values of a single type. In JSON Schema, enumerations are constraints applied to schemas that are not constrained to the declared type of the same schema and values can be of mixed types.

There are further issues with the JSON Schema spec like the confusing existence of embedded subschemas or why "vocabularies" for meta-schemas are special-cased and aren't just another schema. This document is not meant to be an exhaustive list of problems.

JSON Schema, as it stands, is a powerful JSON validation language, but a very poor data definition language.

For APIs, databases, LLMs, and code generation, the industry needs a great data definition language that can also be used for validation. The priorities for the vast majority of practical future applications of JSON Schema are upside down from the current state of the spec.

This set of documents, a proposal for a "new JSON Schema", completely refactors JSON Schema into:

a core data definition language that maps from and to code and databases in a straightforward way and also takes typical type reuse and extensibility patterns into account.
a set of optional extensions that provide the powerful composition and validation capabilities that JSON Schema is known for.

In addition, new JSON Schema has a vastly expanded built-in type system that is not limited to the JSON primitives and includes many extended types that are relevant for modern data processing. The type system also directly addresses common pitfalls of the JSON primitives, such as the limited range and precision of numbers.

Optional extensions directly support multi-language documentation and alternate names and descriptions for properties and types as well as annotations for scientific units and currencies based on international standards.

Start with the "primer" document from which this post is an excerpt.

H1/2025 Conference session proposals

Fri, 29 Nov 2024 09:03:50 GMT

I just put a few new session titles/abstracts into my Sessionize speaker profile for the coming year:

1. Bringing Clarity to Event Streams: Enabling Analytics and AI Through Rich Metadata https://sessionize.com/s/clemens-vasters/bringing-clarity-to-event-streams-enabling-analyti/111971

2. Streamifying Reference Data for Temporal Consistency with Telemetry Events https://sessionize.com/s/clemens-vasters/streamifying-reference-data-for-temporal-consisten/111973

3. Real-time data journeys. Kafka, MQTT, AMQP, HTTP, Web Sockets; protocols, my, oh, my! https://sessionize.com/s/clemens-vasters/real-time-data-journeys.-kafka-mqtt-amqp-http-web-/111974

4. From Observations to Actions: Real-Time Intelligence with Microsoft Fabric https://sessionize.com/s/clemens-vasters/from-observations-to-actions-real-time-intelligenc/111982

5. Generating code and projects in all sorts of languages with Python and Jinja2 https://sessionize.com/s/clemens-vasters/generating-code-and-projects-in-all-sorts-of-langu/111972

(It's an unusual topic for me, but I have a ton of material for #5; that'll have to be a 2nd talk at a conference along one of the others, though)

If you run a conference and find any of these abstracts interesting, let me know: clemensv@microsoft.com

"Plain JSON" encoding for Apache Avro

Wed, 13 Nov 2024 12:13:07 GMT

Also available here. JIRA-3986. Illustrating Avro project PR for C#.

The Apache Avro project defines a JSON Encoding, which is optimized for encoding data in JSON, but primarily aimed at exchanging data between implementations of the Apache Avro specification. The choices made for this encoding severely limit the interoperability with other JSON serialization frameworks. This document defines an alternate, additional mode for Avro JSON Encoders, preliminarily named "Plain JSON", that specifically addresses identified interoperability blockers.

While this document is a proposal for a set of new features in Apache Avro, the extensibility of Avro's schema model allows for the implementation of these features separately from the Avro project. Out of the available and popular schema languages for data exchange, Avro schema provides the cleanest foundation for mapping wire representations to programming language types and database tables, which is why interoperability of Avro with the most popular text encoding format for structured data, JSON, is very desirable.

With Avro's strength and focus being its binary encoding, supporting JSON is specifically desireable in interoperability scenarios where either the producer or the consumer of the encoded data is using a different JSON encoding framework, or where JSON is crafted or evaluated directly by the application.

As most JSON document instances can be structurally described by Avro Schema, the interoperability case is for JSON data, described by Avro Schema, to be accepted by an Apache Avro messaging application, and for that data then to be forwarded onwards using Avro binary encoding. Reversely, it needs to be possible for an application to transform an Avro binary encoded data structure into JSON data that is understood by parties that expect to handle JSON. The kinds applications requiring such transformation capabilities are stream processing frameworks, API gateways and (reverse) proxies, and integration brokers.

The intent of this proposal is for the Avro "JsonEncoder" implementations to have a new mode parameter, accepting an enumeration choice out of the options "Avro Json" (AVRO_JSON, AvroJson, etc), which is Avro's default JSON encoding, and "Plain JSON" (PLAIN_JSON, PlainJson, etc). The rules for the "Plain JSON" mode are described herein.

The "Plain JSON" mode is a selector for enabling set of features that are described below. Implementations MAY also choose for these features to be individually selectable for the "Avro JSON" mode, for instance letting the user use the "Avro JSON" mode primarily, but opting into the binary data handling or date-time handling features described here. However, the "Plain JSON" mode that combines these features MUST be implemented to ensure interoperability.

Notational Conventions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Interoperability issues of the Avro JSON Encoding with common JSON usage

There are several distinct issues in the Avro JSON Encoding that cause conflicts with common usage of JSON and many serialization frameworks. It needs to be emphasized that none of these issues are conformance issues with the JSON specification (RFC8259), but rather stem from the JSON specification's inherent limitations. JSON does not define binary data, date or time types. JSON also has no concept of a type-hint for data structures (i.e. objects), which would allow serialization frameworks to establish an unambiguous mapping between a data type in a programming language or schema and the encoded type in JSON.

There are, however, commonly used conventions to address these shortcomings of the core JSON specification:

Binary data: Binary data is commonly encoded using the base64 encoding and stored in string-typed values.
Date and time data: Date and time data is commonly encoded using the RFC3339 profile of ISO8601 and stored in string-typed values.
Type hints: In its native type system, JSON value types are distinguished by notation where 'null' values, strings, numbers, booleans, arrays, and objects are identifiable through the syntax. While JSON has no further data type concepts, several serialization frameworks and even some standards leaning on JSON (e.g. OpenAPI) introduce the notion of a "discriminator" property, which is inside the encoded object and unambiguously identifies the type such that the decoding stage can instantiate and populate the correct type in cases where multiple candidate types exist.

On each of these items, the Avro JSON encoding's choices are in direct conflict with predominant practice:

Binary data: Binary data is encoded in strings using Unicode escape sequences (example: "\u00DE\u00AD\u00BE\u00EF"), which leads to a 500% overhead compared to the encoded bytes vs. a 33% overhead when using Base64.
Date and time data: Avro handles date and time as logical types, extending either long or int, using the UNIX epoch as the baseline. Durations are expressed using a bespoke data structure. As there are no handling rules for logical types in the JSON encoding, the encoded results are therefore epoch numbers without annotations like time zone offsets.
Type-hints: Whenever types can be ambiguous in Avro, which is the case with type unions, the Avro JSON encoding prescribes encoding the value wrapped inside an object with a single property where the property's name is the type name, e.g. "myprop": {"string": "value"}. 'null' values are encoded as 'null', e.g. "myprop": null. For primitive types, this is in conflict with JSON's native type model that already makes the distinction syntactically. For object types (Avro records), the wrapper is in conflict with standing practice where the discriminator is inlined.

In addition, there are three general limitations of Avro's type and schema model that result in potential interoperability blockers:

Avro represents decimal numeric types as a logical type annotating fixed or byte, which results in an encoded byte sequence in the JSON encoding that cannot be interpreted without the Avro schema and is therefore undecipherable for regular JSON consumers.
name fields in Avro are limited to a character set that can be easily mapped to mostly any programming language and database, but JSON object keys are not.
JSON documents may have top-level arrays and maps, while Avro schemas only allow record and enum as independent types and therefore at the top-level of a schema.

As a consequence of this, the current implementations of the Avro JSON Encoding do not interoperate well with "plain JSON" as input and often do not yield useful plain JSON as output. There is a "happy path" on which the Avro JSON Encoding does line up with common usage, but it's easy to stray off from it.

The Plain JSON encoding

The Plain JSON encoding mode of Apache Avro consists of a combination of 7 distinct features that are defined in this section. The design is grounded in the relevant IETF RFCs and provides the broadest interoperability with common usage of JSON, while yet preserving type integrity and precision in all cases where the Avro Schema is known to the decoding party.

The features are designed to be orthogonal and can be implemented separately.

1: Alternate names
2: Avro binary and fixed type data encoding
3: Avro decimal logical type data encoding
4: Avro time, date, and duration logical types
5: Handling unions with primitive type values and enum values
6: Handling unions of record values and of maps
7: Document root records

Features 2, 3, 4, and 5 are trivial on all platforms and frameworks that handle JSON. Features 1 and 7 are hints for the JSON encoder and decoder to be able to handle JSON data that is not conforming to Avro's naming and structure constraints. Feature 6 provides a mechanism to handle unions of record types that is aligned with common JSON encodation frameworks and JSON Schema's "oneOf" type composition.

Feature 1: Alternate names

JSON objects allow for keys with arbitrary unicode strings, with the only restriction being uniqueness of keys within an object. Uniqueness is a "SHOULD" rule in RFC8259, Section 4, which is interpreted as REQUIRED for this specification since it is common practice.

The character set permitted for Avro names is constrained by the regular expression [A-Za-z_][A-Za-z0-9_]*, which poses an interoperability problem with JSON, especially in scenarios where internationalization is a concern. While English is the dominant language in most developer scenarios, metadata might be defined by end-users and in their own language. It's also fairly common for JSON object keys to contain word-separator characters other than '_' and keys may quite well start with a number.

As the Avro project will presumably want to avoid introducing schema attributes that are JSON-specific and will want to use new schema constructs for additional needs as they arise, the alternate names feature introduces a map of alternate names of which the plain JSON feature reserves a key:

`altnames` map

Wherever Avro Schema requires a name field, an altnames map MAY be defined alongside the name field, which provides a map of alternate names. Those names may be local-language identifiers, display names, or names that contain characters disallowed in Avro. The map key identifies the context in which the alternate name is used.

This specification reserves the json key in the altnames map.

A display-name feature might reserve display:{IANA-subtag} as keys. This assumed convention is used in the following example just for illustration of the altnames feature.

Assume the following JSON input document with German-language keys that represents a row in commercial order document:

{
  "Artikelschlüssel": "1234",
  "Stückzahl": 42,
  "Größe": "Extragroß"
}

Without the alternate names feature, the Avro schema would not be able to match the keys in the JSON document since ü and ß are not allowed. With the alternate names feature, the schema can be defined as follows:

{
  "type": "record",
  "namespace": "com.example",
  "name": "Article",
  "fields": [
    {
      "name": "articleKey",
      "type": "string",
      "altnames": {
        "json": "Artikelschlüssel",
        "display:de": "Artikelschlüssel",
        "display:en": "Article Key"
      }
    },
    {
      "name": "quantity",
      "type": "int",
      "altnames": {
        "json": "Stückzahl",
        "display:de": "Stückzahl",
        "display:en": "Quantity"
      }
    },
    {
      "name": "size",
      "type": "sizeEnum",
      "altnames": {
        "json": "Größe",
        "display:de": "Größe",
        "display:en": "Size"
      }
    }
  ]
}

When the JSON decoder (de-)encodes a named item, the encoder MUST use the value from the altnames entry with the json key as the name for the corresponding JSON element, when present.

`altsymbols` map

The altsymbols map is a similar feature to altnames, but it is used for alternate names of enum symbols. The altsymbols map provides alternate names for symbols. As with altnames, the altsymbols map key identifies the context in which the alternate name is used. The values of the altsymbols map are maps where the keys are symbols as defined in the symbols field and the values are the corresponding alternate names.

Any symbol key present in the altsymbols map MUST exist in the symbols field. Symbols in the symbols field MAY be omitted from the altsymbols map.

{
  "type": "enum",
  "name": "sizeEnum",
  "symbols": ["S", "M", "L", "XL"],
  "altsymbols": {
    "json": {
      "S": "Klein",
      "M": "Mittel",
      "L": "Groß",
      "XL": "Extragroß"
    },
    "display:en": {
      "S": "Small",
      "M": "Medium",
      "L": "Large",
      "XL": "Extra Large"
    }
  }
}

When the JSON decoder (de-)encodes an enum symbol, the encoder MUST use the value from the altsymbols entry with the json key as the string representing the enum value, when present.

Feature 2: Avro `binary` and `fixed` type data encoding

When encoding data typed with the Avro binary or fixed types, the byte sequence is encoded into and from Base64 encoded string values, conforming with IETF RFC4648, Section 4.

Feature 3: Avro `decimal` logical type data encoding

When encoding data typed with the Avro logical decimal type, the numeric value is encoded into a from a JSON number value. JSON numbers are represented as text and do not lose precision as IEEE754 floating points do.

When using a JSON library to implement the encoding, decimal values MUST NOT be converted through an IEEE floating point type (e.g. double or float in most programming languages) but must use the native decimal data type.

Feature 4: Avro time, date, and duration logical types

When encoding data typed with one of Avro's logical data types for dates and times, the data is encoded into and from a JSON string value, which is an expression as defined in IETF RFC3339.

Specifically, the logical types are mapped to certain grammar elements defined in RFC3339 as defined in the following table:

logicalType	RFC3339 grammar element
`date`	RFC3339 5.6. “full-date”
`time-millis`	RFC3339 5.6. “date-time”
`time-micros`	RFC3339 5.6. “partial-time”
`timestamp-millis`	RFC3339 5.6 “date-time”
`timestamp-micros`	RFC3339 5.6 “date-time”
`local-timestamp-millis`	RFC3339 5.6 “date-time”, ignoring offset (note RFC 3339 4.4)
`local-timestamp-micros`	RFC3339 5.6 “date-time” , ignoring offset (note RFC 3339 4.4)
`duration`	RFC3339 Appendix A “duration”

Feature 5: Handling unions with primitive type values and enum values

Unions of primitive types and of enum values are handled through JSON values' (RFC8259, Section 3) ability to reflect variable types.

Given a type union of [string, null] and a string value "test", a encoded field named "example" is encoded as "example": null or "example": "test". For null-valued fields, the JSON encoder MAY omit the field entirely. During decoding, missing fields are set to null. If a default value is defined for the field, decoding MUST set the field value to the default value.

For a type union of [string,int] and string values "2" and the int value 2, a encoded field named "example" is encoded as "example": "2" or "example":2.

For a type union of [null, myEnum] with myEnum being an enum type having symbols "test1" and "test2", a encoded field named "example" is encoded as "example": null or "example": "test1" or "example": "test2".

Instances of unions of primitive types with arrays and records or maps can also be distinguished through the JSON grammar and type model. Unions of multiple records are discussed in Feature 6 below.

For completeness, these are the updated type mappings of Avro types to JSON types for the plain JSON encoding.

Avro type	JSON type	Notes
null	null	The field MAY be omitted
boolean	boolean
int,long	integer
float,double	number
bytes	string	Base64 string, see Feature 2
string	string
record	object
enum	string
array	array
map	object
fixed	string	Base64 string, see Feature 2
date/time	string	See Feature 4
UUID	string
decimal	number	See Feature 3

Feature 6: Handling unions of record values and of maps

As discussed in the overview, JSON does not have an inherent concept of a type-hint that allows distinguishing object data types. Indeed, it has no concept of constraining and further specifying the object type, at all.

The JSON Schema project has defined a schema language specifically for JSON data and provides a type concept for object. In JSON interoperability scenarios, JSON Schema, or frameworks that infer their type concepts from JSON Schema, will often play a role on the producer or consumer side due to its popularity.

JSON Schema is primarily a schema model that serves to validate JSON documents. Its "oneOf" type composition construct is equivalent to Avro's union concept in function. Out of a choice of multiple type options, exactly one option MUST match the JSON element that is being validated, otherwise the validation fails. Any implementation of a JSON Schema validator must therefore be able to test the given JSON element against all available options and then determine the matching type option. Any implementation of a schema driven decoder can use the same strategy to select which type to instantiate and populate.

JSON Schema does not define a type-hint for this purpose, but makes it the schema designer's task to create type definitions that are structurally distinct such that the "oneOf" test always yields one of the types when given JSON element instances. Schema designers then occasionally resort to introducing their own type-hints by either defining a discriminator property with a single-value enum or with a const value, where the discriminator property name is the same across the type options, but the values of the enum or const are different. We will lean on this practice in the following.

Type structure matching

Consider the following Avro schema with a type union of two record types:

{
  "type": "record",
  "name": "ContactList",
  "fields": [
    {
      "name": "contacts",
      "type": "array",
      "items": [
        {
          "type": "record",
          "name": "CustomerRecord",
          "fields": [
            {"name": "name", "type": "string"},
            {"name": "age", "type": "int"}
            {"name": "customerId", "type": "string"}
          ]
        },
        {
          "type": "record",
          "name": "EmployeeRecord",
          "fields": [
            {"name": "name", "type": "string"},
            {"name": "age", "type": "int"},
            {"name": "employeeId", "type": "string"}
          ]
        }
      ]
    }
  ]
}

Now consider the following JSON document:

{
  "contacts": [
    {"name": "Alice", "age": 42, "customerId": "1234"},
    {"name": "Bob", "age": 43, "employeeId": "5678"}
  ]
}

We can clearly distinguish the two record types by the presence of the respectively required customerId or employeeId field.

When decoding a type union, the JSON decoder MUST test the JSON element against all available type options. A JSON element matches if it can be correctly and completely decoded given the type-union candidate schema, including all applicable nested or referenced definitions. If more than one of the options matches, decoding MUST fail. The JSON decoder MUST select the type option that matches the JSON element and instantiate and populate the corresponding type.

For performance reasons, it is highly desirable to avoid having to test a JSON element against all possible type options in a union and instead have a single property that can be tested first and short-circuits the type matching process. We discuss that next.

Discriminator property

When we assume the Avro schema to be slightly different, we might end up with an ambiguity that is not as easy to resolve. Let the employeeId and customerId fields be optional in the schema above, both typed as ["string", "null"].

When we now consider the following JSON document, we can't decide on the type and will fail decoding:

{
  "contacts": [
    {"name": "Alice", "age": 42},
    {"name": "Bob", "age": 43}
  ]
}

To resolve this ambiguity, we can introduce a discriminator property that clearly identifies the type of the record.

Instead of introducing a schema attribute that is specific to JSON, we instead introduce a new Avro schema attribute const that defines a constant value for the field it is defined on.

The value of the const field must match the field type. The value of the field MUST always match the const value. During decoding, decoding MUST fail if the field value is not equal to the const value. This rule ensures the function of const as a discriminator. The const field is only allowed on fields of primitive types and enum types.

Consider this Avro schema:

{
  "type": "record",
  "name": "ContactList",
  "fields": [
    {
      "name": "contacts",
      "type": "array",
      "items": [
        {
          "type": "record",
          "name": "CustomerRecord",
          "fields": [
            {"name": "name", "type": "string"},
            {"name": "age", "type": "int"},
            {"name": "customerId", "type": ["string", "null"]},
            {"name": "type", "type": "string", "const": "customer"}
          ]
        },
        {
          "type": "record",
          "name": "EmployeeRecord",
          "fields": [
            {"name": "name", "type": "string"},
            {"name": "age", "type": "int"},
            {"name": "employeeId", "type": ["string", "null"]},
            {"name": "type", "type": "string", "const": "employee"}
          ]
        }
      ]
    }
  ]
}

The JSON document MUST now include the discriminator:

{
  "contacts": [
    {"name": "Alice", "age": 42, "type": "customer"},
    {"name": "Bob", "age": 43,  "type": "employee"}
  ]
}

The const field MAY otherwise be used for any other purpose. The binary decoder MAY skip encoding and decoding a field with a const attribute and instead always return the constant value for the field similar to how the default field is handled. The const value overrides the default value. During encoding, the binary encoder SHOULD check that the field value matches the const value and MAY fail encoding if it does not.

Feature 7: Document root records

Avro schemas are defined as a single record or enum type at the top level or as a top-level type union. JSON documents, however, may have top-level arrays and maps. Without changing the fundamental Avro schema model, the plain JSON encoding mode uses an annotation on array and map types defined inside record types to allow for top-level arrays and maps in the JSON document.

The annotation is a boolean flag named root that is set to true on one record field's array or map type. The root flag is only defined for array and map types. If the root flag is present and has the value true, the enclosing record type MUST have exactly this one field.

Given a JSON document with a top-level array like this:

[
  {"name": "Alice", "age": 42},
  {"name": "Bob", "age": 43}
]

The Avro schema would be defined as follows:

{
  "type": "record",
  "name": "PersonDocument",
  "fields": [
    {
      "name": "persons",
      "type": {
        "type": "array",
        "root": true,
        "items": {
          "type": "record",
          "name": "PersonRecord",
          "fields": [
            {"name": "name", "type": "string"},
            {"name": "age", "type": "int"}
          ]          
        }
      }
    }
  ]
}

When the JSON decoder encounters a top-level array or map, it MUST match the array or map to the field with the root flag set to true. When the root flag is present on a field, the JSON encoder MUST yield the encoding of the field as the encoding of the entire record. The JSON encoder MUST fail if the root flag is set to true and if there is more than one field in the record.

When such a record type is used as a field type inside another record, it consequently is always represented equivalent to a map or array type in the JSON document.

In type structure matching scenarios, a set root on a map type causes the record type to be a candidate for the type matching of JSON object values. The root flag on an array type causes the record type to be a candidate for the type matching of JSON array values.

The Avro binary encoding is not functionally affected by this feature, but the structural constraint imposed by the root flag MAY be enforced by the encoder.

11/10/2024

Sun, 10 Nov 2024 00:00:00 GMT

Looking forward to the next season of Andor:

https://variety.com/2024/tv/news/andor-season-2-release-date-rogue-one-prequel-1235812153/

Reply in a Reddit thread on legitimate uses of returning "object" in C#

https://bsky.app/profile/clemens.vasters.com/post/3lalmnd4r2c2s

A semantic identifier for occurrences: CloudEvent "type"

Sat, 09 Nov 2024 00:55:33 GMT

In today's xRegistry WG Friday meeting we had a debate about the relationship between the message identifier (messageid) that uniquely identifies a metadata template for a CloudEvent (if that is the chosen envelope format) and the CloudEvent type attribute.

In the message Registry spec we recommend for those two identifiers to be the same; the example below shows this:

"Contoso.MotorVehicle.OilTemperatureThresholdExceeded": {
    "description": "Alarm event for when the oil temperature exceeds a threshold",
    "envelope": "CloudEvents/1.0",
    "envelopemetadata": {
        "id": { "required": true },
        "type": {
            "value": "Contoso.MotorVehicle.OilTemperatureThresholdExceeded",
            "description": "Event raised when oil temperature exceeds a set threshold"
        },
        "source": {
            "type": "uritemplate",
            "description": "source of the event",
            "value": "{vehicleid}/{componentid}"
        },
        "subject": {
            "type": "string",
            "description": "identifier of the temperature alarm"
        },
        "datacontenttype": {
            "value": "application/vnd.apache.avro+plainjson"
        },
        "time": { "required": true }
    },
    "schemaformat": "Avro/1.11.1",
    "schemauri": "#/schemagroups/Contoso.MotorVehicle/schemas/Contoso.MotorVehicle.OilTemperatureThresholdExceededEventData"
}

We believe that is the right choice for 90% of the cases, but it's a recommendation and not a requirement since there are cases where the two identifiers will be different. The discussion below will be interesting to CloudEvent users even if you don't care about xRegistry at the moment.

What does the CloudEvents `type` attribute mean?

The type attribute of a CloudEvent is an identifier of the kind of occurrence, the thing that happened that the event reports about. A database record was added, a river level gauge reading was made, the oil temperature of a motor exceeded a threshold value. The type identifies those semantics.

The type is not coupled to a specific payload schema or schema version or schema format or data encoding. That is why the CloudEvent has the dataschema and datacontenttype attributes to individually qualify these aspects.

The type is, however, supposed to be coupled to stable semantics. That is to say that when you define an event type together with an initial payload, it's allowed for the payload to be refined over time by adding more information (since you can convey this via dataschema) and it's also allowed to encode the information differently (since you can convey this via datacontenttype).

It's not allowed for the semantic essence of event to change under a given type. The event still needs to report about the same kind of occurrence. Any breaking change, omitting formerly required information, is a semantic change and ought not to be permitted within a type. If the semantics change, you must introduce a new type and you may indeed have to report the same occurrence once per any existing and newly introduced type if you make such a choice.

We did not introduce "event type versions" as an easy switch in CloudEvents. We specifically wanted to put up a high bar for breaking changes. Distributed systems are hard enough while being disciplined about sticking to promises made. They turn into chaos if breaking promises is easy. We effectively mandate that a breaking change is signaled by a whole new type.

Now, I work with quite a few developers absolutely obsessed with precision and I know that "semantic identifier of an occurrence" freaks them out as a concept because it's kinda wobbly. But the lack of specificity around the expression of the event semantics in terms of encodings and schemas is intentional because only that enables eventual evolution. Today, JSON encoding is overwhelmingly the favorite way to move event data, but tomorrow the fashion may change. The encoding choice changing, and with it the schema format required to drive that encoding, does not invalidate the semantic of the event. The the oil temperature of the motor still exceeds its threshold value, even if the expression of it changes.

xRegistry message definitions

xRegistry aims to be a catalog for messaging and eventing, organizing endpoints for discovery, organizing data schemas for sharing validation and serialization metadata, and organizing message metadata to declare "contracts" for communication channels.

You can think of a message definition like the one shown above as a set of constraints (or a template) for an event. The definition above defines a specific expression of a CloudEvent of type Contoso.MotorVehicle.OilTemperatureThresholdExceeded. The definition references an Apache Avro schema (not shown) and mandates for the encoding to be "vnd.apache.avro+plainjson" via a constraint of the datacontenttype attribute.

That is, however, just one way how one might to express this event in a message. You could have an alternate representation using Protobuf3.

"Contoso.MotorVehicle.OilTemperatureThresholdExceeded.Proto3": {
    "description": "Alarm event for when the oil temperature exceeds a threshold",
    "envelope": "CloudEvents/1.0",
    "envelopemetadata": {
        "id": { "required": true },
        "type": {
            "value": "Contoso.MotorVehicle.OilTemperatureThresholdExceeded",
            "description": "Event raised when oil temperature exceeds a set threshold"
        },
        "source": {
            "type": "uritemplate",
            "description": "source of the event",
            "value": "{vehicleid}/{componentid}"
        },
        "subject": {
            "type": "string",
            "description": "identifier of the temperature alarm"
        },
        "datacontenttype": {
            "value": "application/vnd.google.protobuf.v3"
        },
        "time": { "required": true }
    },
    "schemaformat": "Protobuf/3",
    "schemauri": "#/schemagroups/Contoso.MotorVehicle/schemas/Contoso.MotorVehicle.OilTemperatureThresholdExceededEventData.Proto3"
}

This definition is semantically identical as it shared the event type. Once you've deserialized the event into memory, the difference between the versions that flowed as Protobuf and the one that flowed as Avro-defined JSON should vanish completely if the schemas are, as they should, expressing semantically identical data structures.

In a constellation where we want to flow events in different encodings, we will therefore have multiple definitions that have distinct message definition identifiers (messageid), but which refer to the same type as they express the same semantics. A basic test for whether the structurally expressed semantics match across the variants of the event is whether the schemas for the respective encodings can be mapped to the same in-memory data structure easily.

The specificity of a message definition in that it ties together several constraints also means that it's not versionable per-se; message types have variants, not versions. Consequently, xRegistry does not version message definitions, different from how it handles data schemas.

If you change the composition of the constraints, you create a new definition. That is the case because if you ever produced a message based on the given set of constraints, another party using the catalog needs to be able to have the definition available to match against, side-by-side with all other variants. If a message arrives with type Contoso.MotorVehicle.OilTemperatureThresholdExceeded and datacontenttype application/vnd.google.protobuf.v3, having both of the definitions above available concurrently allows matching and subsequently processing according to the metadata/constraints of the second (*.Proto3) one.

The application platform layers that deal with dispatching and deserialization or channel selection and serialization will distinguish events by messageid and, in the dispatcher case, pattern-match incoming events against the concurrently available message definitions to find one specific match. The application layers above will distinguish events by their CloudEvents type as all alternate definitions and their alternate wire representations ought to yield compatible events to the application after processing.

I wrote this down primarily to capture today's WG discussion, but I hope you also find this useful as a CloudEvents practitioner.

[Note: A few links in this document don't deep-link to where they should; that's due to a bug in my fork of the blog engine that I'll fix shortly]