Using the system
Below you’ll find instructions on how to run the OpenSanctions software and how to add additional crawlers to the system. Before we dive into that, however, let’s explore some of the concepts underlying the system.
Datasets, sources and collections
OpenSanctions collects data from a variety of sanctions lists and other data providers
and converts it into a common, simple-to-use data model. These data are grouped into
datasets. Some datasets are sources and refer to a data origin (e.g.
the EU sanctions list. Other datasets combine data from multiple sources into a
sanctions, which collects all sanctions entities from multiple
Both source and collection datasets have a metadata definition, stored as a
YAML file in
opensanctions/metadata. Sources also include crawler code to parse
and import the material. This code is usually located in
Entities and targets
The main objective of OpenSanctions is to combine data from multiple sources into a common data model. To this end, the system uses FollowTheMoney (FtM), a data modelling and validation library which defines a set of entity schemata, such as Person, Company, Address or Sanction. FtM-based entities are stored in a local database and then exported to a variety of file formats.
A peculiarity of the data in OpenSanctions is that sources may mention entities that are merely adjacent to a sanctions target, but not themselves sanctioned. To distinguish the sanctioned entities, they are flagged as targets in the database. For most end users that wish to download and use a simple CSV file, chances are that they will want sanctions targets, without the secondary entities in the dataset.
Using the command-line tool
Once you’ve successfully installed OpenSanctions, you can use the built-in command-line tool to run parts of the system:
# Crawl and export the US consolidated list: $ opensanctions run us_ofac_cons # This works for both sources and collections. Running a collection will # crawl all related sources and then export the collection data: $ opensanctions run sanctions # Running without a specified dataset name will default to using the # `all` collection which contains all sources: $ opensanctions run $ opensanctions run all # If you're developing the crawler, you can skip generating the exports and # only run the crawl stage: $ opensanctions crawl us_ofac_cons # Inversely, you can also export a dataset without re-crawling the sources: $ opensanctions export us_ofac_cons # During development you might also want to force delete all data linked # to a source: $ opensanctions clear us_ofac_cons
The available dataset names are determined from the set of metadata YAML files
OPENSANCTIONS_METADATA_PATH (see: configuration).