wiki:BluePrintDeduplication

Version 5 (modified by Dominic König, 11 years ago) ( diff )

--

Data De-duplication Blue Print

We often get duplicate data in a system, especially if we do Bulk Imports from other data sources but also because many users have the tendency to enter new records, rather than reusing existing records.

Process

  1. Identifying Duplicate Records (using the Jaro Winkler Distance)
    • Comparing the fields used in the "represent" of this table is a good start.
    • In order to determine if the records are in fact duplicate, the user should have the option to open up the records and somehow see where they are referred to.
  2. Merging Duplicate Records (see wireframe)
  3. Replacing Duplicate Records (must work with offline instances over sync too)

A complete specifications can be found at http://wiki.sahanafoundation.org/doku.php/foundation:gsoc_kohli:import:duplicates

See also:

Different Processes Identifying Duplicate Records

Some resources may have unique processes for identifying duplicates:

Locations

Identifying duplicate locations really should involve significant use of maps for context of the two point being checked, perhaps fields showing great-circle distance from each other, and if we have hierarchy polygons available, then performing spatial analysis to see if it is the same town in the same region or two towns that share the same name, but are in different regions?

Documents

Document deduping could use SHA1 checksum analysis of the file to detect dupes (e.g. there is a very low probability of two files sharing the same SHA1 hash), and think that an SHA1 hash should be calculated for a document or image file at time of upload.

People

Peoples names may use Soundex, addresses, phone number etc...although be careful!

Current Progress

  • A basic Locations de-duplicator is now in Trunk. (gis/location_duplicates) This tool will replace all references to Location A (Old) to Location B (new) and delete Location A.
    • It does not provide a method for finding duplicate records
    • If will not work across data in multiple (synced) instances

BluePrints

Note: See TracWiki for help on using the wiki.