The deduplication process for decision tables is an automatic, cyclic background process running on the Higson Studio server. Its purpose is to optimize the storage of decision table matrices by detecting and deduplicating (reusing) identical matrices.
Problem It Solves#
Decision tables in Higson Studio may contain identical matrices. In such cases, the same data is stored multiple times — separately for each decision table. The deduplication process detects such situations and eliminates redundancy:
- one decision table is designated as the owner of the matrix,
- while the others become hard links pointing to that owner.
The physical data exists only once, regardless of how many decision tables share it.
Benefits#
- Lower memory (heap) – the DT lookup index is reused whenever possible
- Database storage – the matrix is stored only once in the database
- Smaller snapshot size – DTs that are hard links do not contain matrix files (
*.mat) - Faster DT loading time by the engine (no need to rebuild the lookup index if another DT uses the same matrix)
- Faster snapshot export time
- Faster export time for the list of decision tables
Functionality#
In addition to the deduplication process, hard links may also be created in other scenarios. For example, attaching a global decision table (DT) to multiple versions no longer requires creating deep copies of the DT. Only one DT will contain the matrix, while the others will become hard links.
Configuration#
Configuration is available via four properties:
| Property | Description | Default value |
|---|---|---|
| higson.studio.deduplication.enabled | Enables or disables the hard link feature | false |
| higson.studio.deduplication.job-enabled | Enables or disables the background deduplication process | false |
| higson.studio.deduplication.job-interval | Interval between consecutive runs of the deduplication process (if enabled). Format: number followed by a unit (s, m, h, d) |
30m |
| higson.studio.deduplication.matrix.min-size | The minimum number of matrix rows required for a matrix to qualify for deduplication by the background process |