Metadata
Metadata is a powerful feature that allows you to attach arbitrary custom data to platform resources such as bots, datasets, conversations, and more. This extensibility mechanism enables you to store integration-specific information, tracking data, or custom properties without requiring changes to the platform's core data model.
The metadata system is particularly valuable for building integrations and custom workflows. You can use metadata to store external system identifiers, track the source of data, maintain audit trails, or attach any other information your application needs. Many customers rely on metadata to make decisions and provide feedback to users in their systems.
How Metadata Merging Works
When updating resources, the platform provides two distinct approaches for managing metadata: full replacement and incremental updates. Understanding these patterns is essential for correctly managing custom data throughout your resource lifecycle.
Full Replacement Mode replaces the entire metadata object with new data.
When you provide a metadata object directly without the special $update
key, the system completely replaces any existing metadata with your new
values. This approach is useful when you want to reset metadata or when
you're managing the complete metadata state in your application.
In full replacement mode, any previous metadata keys not included in the update request are removed. This gives you complete control over the metadata state but requires you to provide all desired fields on each update.
Incremental Update Mode merges new metadata with existing data,
preserving fields not included in the update. This mode is activated by
wrapping your metadata updates in a special $update object. The platform
merges your new values with existing metadata, only overwriting the specific
fields you provide while preserving all other existing metadata.
With incremental updates, if the resource previously had metadata like
{"departmentId": "support", "version": "1.0"}, the result after the update
would be {"departmentId": "support", "version": "1.0", "lastSyncedAt": "2025-01-11T20:00:00Z", "recordCount": 150}.
The existing departmentId and version fields are preserved, while new
fields are added and matching fields are updated.
Metadata Key Filtering
The metadata system automatically filters out keys that begin with non-word characters such as underscores, hyphens, or special symbols. This filtering applies in both full replacement and incremental update modes, ensuring that internal or temporary keys are not persisted to the database.
After processing, only validKey and anotherValid will be stored in the
resource's metadata. Keys starting with _, -, or other non-word
characters are automatically removed. This behavior protects against
accidentally persisting temporary or internal data that should not be part
of the permanent metadata record.
Important Considerations
Backward Compatibility Warning: Many customers rely on specific metadata structures to make decisions and provide feedback in their integrated systems. Changes to metadata structure or the merging behavior could break existing integrations. Always maintain backward compatibility when adding new features or modifying how metadata is processed.
Performance and Size: While metadata is flexible, be mindful of the size of data you store. Large metadata objects can impact query performance and increase storage costs. Consider storing references to external data rather than embedding large datasets directly in metadata.
Data Privacy: Metadata is subject to the same access controls as the resource it's attached to. However, ensure you're not storing sensitive information in metadata unless it's properly protected and your security requirements are met.
Use Cases: Common metadata use cases include storing external system IDs for integration purposes, tracking data lineage and provenance, maintaining custom workflow states, attaching version information, storing UI preferences, and tracking last synchronization times for external data sources.