name: implement-schema-evolution description: Implement schema evolution features including semantic versioning, migration planning, breaking change detection, and schema history tracking license: MIT compatibility: opencode,claude metadata: created_at: "2026-01-29T22:55:16.161Z" updated_at: "2026-01-29T22:55:16.161Z" version: "1" tags: "transformation,schema,versioning,migration" complexity: "high" dependencies: "vibe_piper.types, vibe_piper.decorators" stability: "stable" examples: "Semantic versioning, schema migration, breaking change detection"

Implement Schema Evolution for Vibe Piper

This skill provides comprehensive schema evolution capabilities for data pipelines.

When To Use

• User requests schema versioning, migration, or evolution features
• Ticket requires schema evolution implementation
• Need to track schema versions and changes over time

Implementation Guide

Core Components

1. SemanticVersion: Semantic versioning (MAJOR.MINOR.PATCH)

   from vibe_piper.schema_evolution import SemanticVersion
   
   v = SemanticVersion(1, 2, 3)  # 1.2.3
   v.is_compatible()  # Check version compatibility
   v.next_major()  # Increment major version
   

2. SchemaDiff: Compare schemas and detect changes

   from vibe_piper.schema_evolution import BreakingChangeDetector, SchemaDiff
   
   detector = BreakingChangeDetector()
   diff = detector.detect(old_schema, new_schema)
   diff.changes  # List of SchemaChange objects
   diff.has_breaking_changes  # Boolean
   

3. MigrationPlan: Generate and execute migration plans

   from vibe_piper.schema_evolution import MigrationPlanner, MigrationPlan
   
   planner = MigrationPlanner()
   plan = planner.generate(diff)
   migrated_data = plan.execute(old_data)
   

4. SchemaHistory: Track schema versions over time

   from vibe_piper.schema_evolution import SchemaHistory, get_schema_history
   
   history = get_schema_history()
   history.add_entry(entry)
   

5. @schema_version decorator: Declarative versioning

   from vibe_piper import define_schema, schema_version
   
   @define_schema
   @schema_version("1.2.0", description="Add email field")
   class UserSchema:
       email: String = String(max_length=255)
   

Key Patterns

• Breaking changes are detected when:

  • Fields are removed
  • Field types change (e.g., INT to STRING)
  • Fields become required (optional → required)
  • Fields become non-nullable (nullable → not nullable)
  • Constraints tighten (e.g., max_value decreases)

• Non-breaking changes:

  • Fields are added
  • Fields become optional (required → optional)
  • Fields become nullable (not nullable → nullable)
  • Constraints relax

• Migration steps are auto-generated based on schema diffs

• Custom migration functions can be provided for complex transformations

Testing Strategy

import pytest
from vibe_piper import SemanticVersion, define_schema, schema_version

def test_schema_evolution():
    # Test version parsing and comparison
    v1 = SemanticVersion.parse("1.2.0")
    v2 = SemanticVersion(1, 3, 0)
    assert v2 > v1

    # Test breaking change detection
    detector = BreakingChangeDetector()
    diff = detector.detect(old_schema, new_schema)
    assert diff.has_breaking_changes

    # Test migration plan execution
    plan = MigrationPlanner().generate(diff)
    migrated = plan.execute(old_data)
    ```

## Acceptance Criteria
- [ ] @schema_version decorator working
- [ ] Automatic migration when schema changes
- [ ] Backward compatibility validation
- [ ] Breaking change warnings
- [ ] Schema diff visualization (MigrationPlan.to_dict)
- [ ] Migration plan generation
- [ ] Rollback to previous version (SchemaHistory.rollback_to_version)
- [ ] Schema history in metadata store (needs integration)
- [ ] Integration with @asset decorator (works but needs verification)
- [ ] Tests with schema evolution scenarios
- [ ] 80%+ coverage
- [ ] Documentation

## Dependencies

- vibe_piper.types (Schema, SchemaField, DataType)
- vibe_piper.schema_definitions (define_schema, Field types)
- vibe_piper.decorators (@asset decorator)
- Python 3.14+ (for type annotations)

## Notes

- This is a core transformation feature that integrates with @asset decorator
- Schema history is currently in-memory; metadata store integration is a future enhancement
- MigrationPlan.to_dict() provides JSON serialization of migration steps
- Rollback functionality is available but requires data backup strategy
<!-- END:compound:skill-managed -->

## Manual notes

_This section is preserved when the skill is updated. Put human notes, caveats, and exceptions here._