Retract tuple, relation and clear relation (truncate) (#9)

2 files changed, 512 insertions, 65 deletions

diff --git a/docs/code.org b/docs/code.org
new file mode 100644
index 0000000..30ff9c9
--- /dev/null
+++ b/docs/code.org
@@ -0,0 +1,272 @@
+#+TITLE: Code Documentation
+
+* Overview
+
+Domino is a content-addressed, immutable relational database engine written in Erlang. The system uses cryptographic hashing (SHA-256) and Merkle trees to provide version control, deduplication, and efficient synchronization capabilities.
+
+* Architecture
+
+** Design Patterns
+
+- *Content-Addressed Storage* :: All data (attributes, tuples, relations, databases) are identified by their cryptographic hash
+- *Immutable Data Structures* :: All operations create new versions rather than modifying existing data
+- *Merkle Trees* :: Relations and databases use merkle trees for efficient diffing and synchronization
+- *Volcano Iterator Model* :: Query execution uses lazy evaluation with process-based iterators
+- *Actor Model* :: Iterators are implemented as independent Erlang processes
+- *Transactional Storage* :: Mnesia provides ACID guarantees for all mutations
+
+* Module Structure
+
+** operations.erl
+
+The core database operations module and main API.
+
+*** Public API
+
+**** Database & Relation Management
+- =setup/0= :: Initialize Mnesia schema and create all tables
+- =create_database/1= :: Create a new database state
+- =create_relation/3= :: Add a relation to a database with schema
+- =get_relations/1= :: List all relation names in a database
+- =get_relation_hash/2= :: Get relation hash by name
+
+**** Tuple Operations
+- =create_tuple/3= :: Insert a tuple into a relation (transactional)
+- =retract_tuple/3= :: Remove a tuple from a relation (default: transactional)
+- =retract_tuple/4= :: Remove a tuple with optional transaction control
+
+**** Relation Operations
+- =clear_relation/2= :: Remove all tuples from a relation (default: transactional)
+- =clear_relation/3= :: Clear relation with optional transaction control
+- =retract_relation/2= :: Remove a relation from database (default: transactional)
+- =retract_relation/3= :: Remove relation with optional transaction control
+
+**** Iterator Pattern (Volcano Model)
+- =get_tuples_iterator/2= :: Create lazy iterator process for relation tuples
+- =next_tuple/1= :: Pull next tuple from iterator (returns ={ok, Tuple} | done=)
+- =close_iterator/1= :: Terminate iterator process
+- =collect_all/1= :: Helper to consume entire iterator into list
+
+**** Utility Functions
+- =hash/1= :: SHA-256 hash of term
+- =hashes_from_tuple/1= :: Extract all tuple hashes from a relation
+
+** relation.erl
+
+Simple helper module for building tuples and relations conceptually.
+
+*** Public API
+- =new_attribute/2= :: Create attribute map: =#{name => Name, value => Value}=
+- =new_tuple/1= :: Convert list of attributes to map
+- =new_relation/1= :: Convert list of tuples to sets:set
+
+** poc.erl
+
+Proof-of-concept demonstrating the Volcano Iterator Model for query execution.
+
+*** Key Concepts
+- Content-addressed storage simulation using in-memory maps
+- Lazy iterator chain: Scan → Select → Project
+- Message-passing protocol for tuple streaming
+- Automatic deduplication via hashing
+
+** main.erl
+
+OTP application entry point. Starts Mnesia on application startup.
+
+* Data Structures
+
+All core records are defined in =include/operations.hrl=.
+
+** attribute
+#+BEGIN_SRC erlang
+-record(attribute, {
+    hash,    % SHA-256 hash of value
+    value    % Actual value (any Erlang term)
+}).
+#+END_SRC
+
+** tuple
+#+BEGIN_SRC erlang
+-record(tuple, {
+    hash,           % SHA-256 hash of entire tuple
+    relation,       % Relation name (atom)
+    attribute_map   % #{attr_name => value_hash}
+}).
+#+END_SRC
+
+** relation
+#+BEGIN_SRC erlang
+-record(relation, {
+    hash,    % hash({name, schema, tree_root})
+    name,    % Relation name (atom)
+    tree,    % Merkle tree of tuple hashes
+    schema   % Schema definition map
+}).
+#+END_SRC
+
+** database_state
+#+BEGIN_SRC erlang
+-record(database_state, {
+    hash,       % Merkle tree root hash
+    name,       % Database name (atom)
+    tree,       % Merkle tree of relations
+    relations,  % #{relation_name => relation_hash}
+    timestamp   % Creation/modification timestamp
+}).
+#+END_SRC
+
+* Data Flow
+
+** Creating a Tuple
+
+1. Store each attribute value separately (enables deduplication)
+2. Build attribute_map with hashes: =#{name => AttrHash1, age => AttrHash2}=
+3. Create tuple record with hash
+4. Insert tuple hash into relation's merkle tree
+5. Compute new relation hash from tree root
+6. Update database merkle tree
+7. Return updated database and relation
+
+All operations occur within a Mnesia transaction.
+
+** Querying Tuples
+
+1. Lookup relation hash from database
+2. Read relation from Mnesia
+3. Extract tuple hashes from merkle tree
+4. Spawn iterator process with hash list
+5. Client pulls tuples via message passing
+6. Iterator resolves: hash → tuple → attribute hashes → actual values
+7. Return materialized tuple map
+
+* Usage Examples
+
+** Basic Workflow
+
+#+BEGIN_SRC erlang
+%% Setup
+operations:setup().
+
+%% Create database
+DB = operations:create_database(my_app).
+
+%% Create relation with schema
+{DB1, _Rel} = operations:create_relation(DB, users, #{
+    name => string,
+    age => integer
+}).
+
+%% Insert tuples
+{DB2, _} = operations:create_tuple(DB1, users, #{name => "Alice", age => 30}).
+{DB3, _} = operations:create_tuple(DB2, users, #{name => "Bob", age => 25}).
+
+%% Query tuples
+Iterator = operations:get_tuples_iterator(DB3, users).
+Tuples = operations:collect_all(Iterator).
+%% Returns: [#{name => "Alice", age => 30}, #{name => "Bob", age => 25}]
+
+%% Remove tuple
+{ok, RelHash} = operations:get_relation_hash(DB3, users).
+[TupleHash|_] = operations:hashes_from_tuple(RelHash).
+{atomic, {DB4, _}} = operations:retract_tuple(DB3, RelHash, TupleHash).
+
+%% Clear relation (remove all tuples, keep schema)
+{atomic, {DB5, _}} = operations:clear_relation(DB4, RelHash).
+
+%% Remove relation entirely
+{atomic, DB6} = operations:retract_relation(DB5, users).
+#+END_SRC
+
+** Iterator Protocol
+
+#+BEGIN_SRC erlang
+Iterator = operations:get_tuples_iterator(DB, users).
+
+%% Pull tuples one by one
+case operations:next_tuple(Iterator) of
+    {ok, Tuple} -> process_tuple(Tuple);
+    done -> io:format("No more tuples~n")
+end.
+
+%% Or collect all at once
+AllTuples = operations:collect_all(Iterator).
+
+%% Clean up
+operations:close_iterator(Iterator).
+#+END_SRC
+
+* Key Implementation Details
+
+** Immutability & Versioning
+
+Every modification creates a new hash. Old versions remain accessible via their hash. Database evolution is tracked via timestamp and hash chain.
+
+** Deduplication
+
+Identical attribute values are stored once (content-addressed). Example: 1000 users with age=30 store the value "30" only once.
+
+** Transaction Semantics
+
+All mutations are wrapped in =mnesia:transaction/1=. Dirty reads are used for immutable data lookups.
+
+** Merkle Tree Strategy
+
+- Relations store tuple hashes as keys: ={TupleHash, <<>>}=
+- Databases store relation names → hashes: ={RelationName, RelationHash}=
+- Enables efficient diff computation between versions
+- Foundation for distributed synchronization
+
+** Iterator Pattern
+
+- Decouples query execution from data access
+- Enables pipelined query operators (future: select, project, join)
+- Memory bounded (doesn't materialize entire relation)
+- Process-based concurrency via Erlang actors
+
+** Schema Flexibility
+
+Schema stored as map (e.g., =#{name => string, age => integer}=). Currently not enforced, but provides foundation for future domain validation.
+
+* Dependencies
+
+- =merklet= (2.0.0) :: Merkle tree implementation for set reconciliation
+- =mnesia= :: Distributed database for persistent storage
+- =crypto= :: SHA-256 hashing
+
+* Testing
+
+The test suite (=test/operations_test.erl=) validates:
+
+- Functional correctness (CRUD operations)
+- Hash determinism and consistency
+- Transactional integrity
+- Iterator protocol compliance
+- State evolution tracking
+- Cross-database consistency
+
+Run tests with:
+
+#+BEGIN_SRC sh
+rebar3 eunit
+#+END_SRC
+
+* Current State
+
+** Implemented
+- Core data model (attribute, tuple, relation, database)
+- Content-addressed storage with SHA-256
+- Immutable versioning
+- Merkle tree integration
+- Iterator-based lazy evaluation
+- Transactional CRUD operations
+- Comprehensive test suite
+
+** Not Yet Implemented
+- Relational algebra operators (σ, π, ⋈, ∪, ∩, −)
+- Schema validation and type checking
+- Infinite relations (aleph₀)
+- Join indexing
+- Query optimization
+- Distributed synchronization
+- REPL/query console
diff --git a/docs/features.org b/docs/features.org
index a6e592e..945bfaf 100644
--- a/docs/features.org
+++ b/docs/features.org
@@ -1,96 +1,271 @@
-* Relational Algebra Engine
+#+TITLE: Domino Features & Roadmap
+
+* Extended Relational Model (RM/T)
   :PROPERTIES:
-  :CATEGORY: relational-algebra
+  :CATEGORY: semantic-modeling
   :END:
-  A pure Erlang implementation of E. F. Codd’s relational algebra.
-  Relations are treated as mathematical sets of tuples; operations are defined exactly
-  as in relational theory, with optional aleph₀ (infinite) relations.
+  An extended relational database engine based on E. F. Codd's RM/T (1979).
+  Captures more meaning through entity types, surrogates, and molecular semantics.
+
+** Implemented Features
+
+*** DONE Content-Addressed Storage
+    CLOSED: [2025-01-21]
+    - [X] SHA-256 surrogates for all entities
+    - [X] System-controlled entity identifiers (E-domain)
+    - [X] Automatic deduplication via content addressing
+    - [X] Immutable data structures
+
+*** DONE Entity Types (E-Relations)
+    CLOSED: [2025-01-21]
+    - [X] Kernel entity types (database, relation)
+    - [X] E-relations for entity existence assertion
+    - [X] Surrogate-based entity identification
+    - [X] Entity integrity enforcement
+
+*** DONE Property Relations (P-Relations)
+    CLOSED: [2025-01-21]
+    - [X] Immediate property storage
+    - [X] Single-valued property support
+    - [X] Property integrity (Rule 4)
+    - [X] Foreign key references via surrogates
+
+*** DONE Merkle Tree Integration
+    CLOSED: [2025-01-21]
+    - [X] Relations as Merkle trees
+    - [X] Database state versioning
+    - [X] Efficient state diffing
+    - [X] Content-addressed relation hashing
+
+*** DONE Volcano Iterator Model
+    CLOSED: [2025-01-21]
+    - [X] Lazy tuple evaluation
+    - [X] Process-based iterators
+    - [X] Message-passing protocol
+    - [X] Memory-bounded queries
+
+*** DONE Transactional Operations
+    CLOSED: [2025-01-21]
+    - [X] ACID guarantees via Mnesia
+    - [X] Atomic tuple insertion
+    - [X] Tuple retraction with transactions
+    - [X] Relation clear and retract operations
+
+*** DONE Comprehensive Test Suite
+    CLOSED: [2025-01-21]
+    - [X] Database creation tests
+    - [X] Relation and tuple operations
+    - [X] Hash consistency validation
+    - [X] Iterator protocol tests
+    - [X] Retraction and truncate tests
+
+** Planned RM/T Extensions
+
+*** TODO Characteristic Entities
+    Multi-valued properties as subordinate entities.
+    - [ ] Characteristic entity types (C-relations)
+    - [ ] Characteristic graph (CG-relation)
+    - [ ] Characteristic tree hierarchy
+    - [ ] Characteristic integrity (Rule 5)
+    - [ ] Example: Job history as characteristic of Employee
+
+*** TODO Associative Entities
+    Relationships as first-class entities with properties.
+    - [ ] Associative entity types (A-relations)
+    - [ ] Association graph (AG-relation)
+    - [ ] Association integrity (Rule 6)
+    - [ ] Higher-order associations
+    - [ ] Example: Assignment(employee, project, start_date)
+
+*** TODO Generalization Hierarchies
+    Entity subtyping and inheritance.
+    - [ ] Unconditional gen inclusion (UGI-relation)
+    - [ ] Alternative gen inclusion (AGI-relation)
+    - [ ] Subtype integrity (Rule 7)
+    - [ ] Property inheritance
+    - [ ] Multiple inheritance support
+    - [ ] Example: Engineer/Secretary/Trucker subtypes of Employee
 
-** Core Specification
-   The theoretical backbone of the system.
+*** TODO Cover Aggregation
+    Collections and groupings.
+    - [ ] Cover membership relation (KG-relation)
+    - [ ] Cover entity types
+    - [ ] Dynamic membership
+    - [ ] Heterogeneous covers
+    - [ ] Example: Convoys, Teams, Departments
 
-*** TODO Define relation as a strict mathematical set
-    - [ ] Each tuple is a set of attributes (no duplicates, no ordering)
-    - [ ] Each attribute = (name, value) pair
-    - [ ] Attribute domains are strongly typed
+*** TODO Event Precedence
+    Temporal ordering and scripts.
+    - [ ] Event entity types (T-relations)
+    - [ ] Unconditional successor (US-relation)
+    - [ ] Alternative successor (AS-relation)
+    - [ ] Unconditional predecessor (UP-relation)
+    - [ ] Alternative predecessor (AP-relation)
+    - [ ] Example: Order → Shipment → Invoice
 
-*** TODO Enforce "Codd's axioms" in implementation
-    - [ ] No duplicate tuples in any relation
-    - [ ] No implicit ordering of tuples or attributes
-    - [ ] All operations closed over the relation type
-    - [ ] Proper domain checking in all operations
+*** TODO Extended Operators
+    Operators for molecular semantics.
+    - [ ] OUTER NATURAL JOIN
+    - [ ] OUTER UNION
+    - [ ] OUTER THETA-JOIN
+    - [ ] MAYBE operators (null handling)
+    - [ ] Graph operators (OPEN, CLOSE, STEP)
 
-** Data Model Layer (`relation.erl`)
-   Fundamental constructs of the relational model.
+*** TODO RM/T Catalog
+    Extensible schema catalog.
+    - [ ] CATR (relations catalog)
+    - [ ] CATRA (relation-attribute)
+    - [ ] CATA (attributes catalog)
+    - [ ] CATAD (attribute-domain)
+    - [ ] CATD (domains catalog)
+    - [ ] CATC (categories catalog)
+    - [ ] CATRC (relation-category)
 
-*** TODO Implement attributes, tuples, relations, and schemas
-*** TODO Add schema validation for domain and arity
-*** TODO Add conversion helpers:
-    - [ ] tuple <-> map
-    - [ ] schema validation checks
+*** TODO Property Graph (PG-relation)
+    Link P-relations to E-relations.
+    - [ ] Property molecule types
+    - [ ] Dynamic property schema
+    - [ ] Property graph traversal
 
-** Algebraic Operations (`operations.erl`)
-   The complete set of classical relational algebra operators.
+** Classical Relational Algebra
 
 *** TODO Selection (σ)
+    Filter tuples by predicate.
+    - [ ] THETA-SELECT with comparison operators
+    - [ ] Support for <, ≤, =, ≥, >, ≠
+    - [ ] TRUE and MAYBE versions (null handling)
+
 *** TODO Projection (π)
+    Project relation onto subset of attributes.
+    - [ ] Remove specified columns
+    - [ ] Eliminate duplicate rows
+    - [ ] Preserve set semantics
+
 *** TODO Cartesian Product (×)
+    Cross product of two relations.
+    - [ ] Unrestricted concatenation
+    - [ ] Basis for joins
+
 *** TODO Natural Join (⋈)
+    Join on common attributes.
+    - [ ] Equi-join with duplicate removal
+    - [ ] Used in normalization
+
 *** TODO Theta Join (⋈θ)
+    Join with arbitrary comparison.
+    - [ ] Support all theta operators
+    - [ ] Generalizes natural join
+
 *** TODO Union (∪)
+    Set union of compatible relations.
+    - [ ] Union-compatible constraint
+    - [ ] Duplicate removal
+
 *** TODO Intersection (∩)
+    Set intersection.
+    - [ ] Union-compatible constraint
+
 *** TODO Difference (−)
+    Set difference.
+    - [ ] Union-compatible constraint
+
+*** TODO Divide (÷)
+    Relational division.
+    - [ ] Algebraic universal quantifier
+
 *** TODO Rename (ρ)
-*** TODO Set semantics enforcement (remove duplicates after each op)
+    Attribute renaming.
+    - [ ] Enable self-joins
+
+** Advanced Features
+
+*** TODO Null Value Handling
+    Three-valued logic for incomplete information.
+    - [ ] "Value unknown" null type
+    - [ ] "Property inapplicable" null type
+    - [ ] E-null for "entity unknown"
+    - [ ] Null substitution principle
+    - [ ] MAYBE operators
+
+*** TODO Infinite Relations (Aleph₀)
+    Countably infinite relation generators.
+    - [ ] Integer generator
+    - [ ] Arithmetic relations (plus, minus, multiply)
+    - [ ] Lazy evaluation with constraints
+    - [ ] Partially materialized storage
+    - [ ] User-defined generators
 
-** Infinite Relations (`aleph0.erl`)
-   Representation of aleph₀-relations (countably infinite generators).
+*** TODO View Integration
+    Merging multiple perspectives.
+    - [ ] View definition language
+    - [ ] Automatic view maintenance
+    - [ ] View update translation
 
-*** TODO Implement integer() generator
-*** TODO Implement plus(), minus(), multiply(), neg() relations
-*** TODO Add constraints and modes for partially bound arguments
-*** TODO Implement lazy generation using streams or coroutines
-*** TODO Support user-defined infinite relations
+*** TODO Temporal Support
+    Time-varying databases.
+    - [ ] Transaction time
+    - [ ] Valid time
+    - [ ] Bitemporal support
+    - [ ] Non-forgetting mode
 
-** Unified Relation Abstraction (`unified_relation.erl`)
-   A bridge between finite and infinite relations.
+*** TODO Distributed Synchronization
+    Merkle tree-based replication.
+    - [ ] Efficient diff protocol
+    - [ ] Conflict resolution
+    - [ ] Eventual consistency
 
-*** TODO Implement unified type for finite and aleph₀
-*** TODO Implement unified select, project, join, union, etc.
-*** TODO Support type checking between finite and infinite relations
-*** TODO Add automatic materialization with configurable limits
-*** TODO Implement caching/memoization for aleph₀ evaluations
+** Developer Experience
 
-** Backend Integration (Mnesia)
-   Persistent storage and retrieval of finite relations.
+*** TODO Query Language
+    High-level relational query language.
+    - [ ] Algebraic syntax
+    - [ ] Tuple calculus variant
+    - [ ] Domain calculus variant
 
-*** TODO Map finite relations to Mnesia tables
-*** TODO Support schema registration and domain enforcement
-*** TODO Add persistence for partially materialized aleph₀ relations
-*** TODO Add atomic transactional updates
-*** TODO Add indexing support for joins
+*** TODO REPL Console
+    Interactive query environment.
+    - [ ] Live query evaluation
+    - [ ] Schema introspection
+    - [ ] Query history
 
-** Correctness and Validation
-   Ensuring theoretical fidelity.
+*** TODO Visualization Tools
+    Graphical representation.
+    - [ ] Relation browser
+    - [ ] Join graph visualization
+    - [ ] Generalization hierarchy diagrams
+    - [ ] Query execution plans
 
-*** TODO Implement relational algebra test suite
-*** TODO Verify closure and idempotence of all operations
-*** TODO Add property-based tests with PropEr or QuickCheck
-*** TODO Compare semantics with formal definitions (e.g., Date & Darwen)
+*** TODO Org Babel Integration
+    Literate programming support.
+    - [ ] Execute relational queries in org-mode
+    - [ ] Export results to tables
+    - [ ] Reproducible analysis
 
-** Developer Tools
-   Optional but desirable tools for exploration and testing.
+** Documentation & Validation
 
-*** TODO Add REPL-like console for interactive relational queries
-*** TODO Add pretty-printer for relations and schemas
-*** TODO Add visualization of join graphs and operation pipelines
-*** TODO Add Org Babel integration for relational queries
+*** DONE Code Documentation
+    CLOSED: [2025-01-21]
+    - [X] Module architecture overview
+    - [X] Data structure specifications
+    - [X] Usage examples
+    - [X] Implementation details
 
-** Documentation
-   Literate and theoretical exposition.
+*** TODO Theoretical Validation
+    Ensure correctness.
+    - [ ] Formal proofs of operator properties
+    - [ ] Closure verification
+    - [ ] Idempotence tests
+    - [ ] Property-based testing with PropEr
+    - [ ] Comparison with Date & Darwen semantics
 
-*** TODO Write README with theoretical introduction
-*** TODO Add module-level comments summarizing relational definitions
-*** TODO Include examples of finite and aleph₀ joins
-*** TODO Provide link to E. F. Codd’s original papers
+*** TODO Research Papers
+    Link to foundational theory.
+    - [ ] Codd 1970: "A Relational Model"
+    - [ ] Codd 1979: "Extending the Database Relational Model" (RM/T)
+    - [ ] Codd 1985: "Does AlOL Need the Relational Algebra?"
+    - [ ] Date & Darwen: "Foundation for Future Database Systems"
 
+** References
+   - E. F. Codd (1979). "Extending the Database Relational Model to Capture More Meaning"
+   - E. F. Codd (1970). "A Relational Model of Data for Large Shared Data Banks"
+   - C. J. Date & Hugh Darwen. "Foundation for Future Database Systems: The Third Manifesto"