Avoid Duplicate Operations allows UnifyApps data pipelines to prevent redundant processing of the same records when implementing cyclical or looped pipeline architectures. This setting creates unique hashes of records and actions to maintain data integrity and prevent duplicative operations during migration processes.
Duplicate Prevention for Cyclical Pipeline Architectures
When configured, UnifyApps implements a hash-based verification system that enables:
One-way data flow even in bidirectional pipeline configurations
Record-level duplicate detection for high-precision control
Resource optimization by preventing redundant processing
Configuring Avoid Duplicate Operations in Pipeline Settings
To enable duplicate prevention in your UnifyApps data pipeline:
Go to the
Settingstab.Under the
Avoid Duplicate Operations (Optional)section, check the box to enable the feature.Save your pipeline configuration.
How Duplicate Prevention Works: Example
Let's walk through a simple example to demonstrate how the duplicate prevention works during data synchronization:
Example: Bidirectional Synchronization Between Systems
Initial Configuration: Day 1
Oracle Database A contains customer records
PostgreSQL Database B needs to maintain synchronized customer data
Pipeline 1: Oracle → PostgreSQL
Pipeline 2: PostgreSQL → Oracle (for updates made in PostgreSQL)
Day 1: Without Duplicate Prevention
Customer record updated in Oracle
Pipeline 1 copies record to PostgreSQL
Pipeline 2 sees "new" record in PostgreSQL
Pipeline 2 copies record back to Oracle
Pipeline 1 sees "updated" record in Oracle
...infinite loop continues...
Day 1: With Duplicate Prevention Enabled
Customer record updated in Oracle
Pipeline 1 creates hash of record+operation and copies to PostgreSQL
Pipeline 2 detects matching hash for record+operation
Pipeline 2 skips processing this record
Loop terminates properly
Day 3: Data Changes in Both Systems
Two days later, records are updated independently in both systems:
Updated Data:
Oracle: Customer #1001 phone updated to 555-1234
PostgreSQL: Customer #1002 address updated to "123 Main St"
Day 3: Synchronization with Duplicate Prevention
Pipeline 1 runs:
- Processes Customer #1001 changes, creates hash, updates PostgreSQL
- Detects Customer #1002 was already processed (has hash), skips
Pipeline 2 runs:
- Processes Customer #1002 changes, creates hash, updates Oracle
- Detects Customer #1001 was already processed (has hash), skips
Notice the key behaviors:
Each record change is processed exactly once
Changes flow properly in both directions
Duplicate processing is avoided through hash verification
Synchronization completes without infinite loops
Practical Use Cases for Avoid Duplicate Operations
Multi-System Data Synchronization When maintaining data consistency across multiple databases:
System A ⟷ System B ⟷ System C
Without duplicate prevention, a change in System A could ping-pong between systems indefinitely.
Change Data Capture with Loopback Verification For CDC processes that include verification workflows:
1. Capture changes in source system
2. Apply to target system
3. Verify changes in target match source
4. Update status in source system
Master Data Management with Multiple Sources of Truth When combining multiple authoritative data sources:
CRM System → MDM Hub ← ERP System
Changes from both systems flow into the hub without creating duplicate updates.
ETL Processes with Validation Loops For complex ETL workflows with validation steps:
Extract → Transform → Load → Validate → Update Source Status
The final status update doesn't trigger re-extraction of the same records.
By implementing the Avoid Duplicate Operations setting, you ensure data integrity across complex pipeline architectures while preventing the resource waste and potential data corruption caused by infinite processing loops. This feature is especially crucial for bidirectional synchronization scenarios or any data pipeline implementation that might create circular data flows.
