Duke is a fast and flexible record linkage engine. It does not use the traditional blocking (sort by key) approach, but instead relies on Lucene. This makes it high-performance (able to process 1,000,000 records in ~10 minutes). Duke can be run from the command line, but also has an API allowing incremental linking applications to be built easily. It supports reading data from CSV, JDBC, SPARQL, and NTriples, and also supports a number of string comparators and string normalizers.
|Tags||Deduplication Record linkage Entity resolution|
|Operating Systems||Java Runtime Environment 1.5|
Release Notes: This release adds much faster backends based on blocking. One is in-memory, the other is based on MapDB. It also has a new Record implementation which uses only 50% of the memory, and a number of other changes.
Release Notes: The main new feature is a genetic algorithm, which can be used to tune configurations automatically. Thanks to active learning it can even be used without a correct set of test data.
Release Notes: Support for multi-threading, an upgrade to Lucene 4.0, higher performance, more comparators, more cleaners, major improvements to the command line client, and more.
Release Notes: New comparators, new cleaners, some bugfixes, an upgrade to Lucene 3.6.1, and some improvements in configurability.
Release Notes: The internals have been cleaned and refactored, adding some performance tuning parameters. There are new cleaners, support for pluggable backends, a new naïve in-memory backend, and much more.