This document describes the process for reporting implemetations of [[PROV-CONSTRAINTS]], i.e. PROV-CONSTRAINTS validators. It provides the instructions for implementers on running PROV-CONSTRAINTS test cases and reporting their results. It also lists all the test cases and links to download them.
The support for constraints from [[PROV-CONSTRAINTS]] in implementations cannot be determined as straightforward as with the features from the other PROV documents as they are, in many cases, interrelated. Sometimes, it is indeed required to consider a number of constraints together to determine the validity of a PROV document. Therefore, in order to facility reporting implementations of PROV-CONSTRAINTS, we provide test cases to assist with determining the level of support of a PROV-CONSTRAINTS implementation.
A test case for PROV-CONSTRAINTS is a PROV document that contains either valid or invalid PROV statements, and implementers of PROV-CONSTRAINTS are asked to determine its validity (as specified in [[PROV-CONSTRAINTS]]). The test cases are specially designed to cover a small set of constraints (called unit test cases, see Section 2.1) or to exercise a subset of PROV statements. By successfully determining the validity of such a test case, an implementation indirectly demonstrates its support for the PROV feature(s) covered by the test case.
Each test case is given an identifier, e.g. ordering-derivation1-PASS-c42, which indicates whether the test case should be successfully validated (i.e. PASS) or not (i.e. FAIL). In addition, a test case's identifier also includes the numberings of the constraints that it covers at the end. e.g. c42.
The provenance document for each test case will be provided in the following representations:
| Representation | File extension | Documentation |
|---|---|---|
| RDF (PROV-O) | .ttl | [[PROV-O]] and [[TURTLE]] |
| PROV-N | .provn | [[PROV-N]] |
| PROV-XML | .provx | [[PROV-XML]] |
For example, the available files for the test case ordering-derivation1-PASS-c42 are: ordering-derivation1-PASS-c42.ttl, ordering-derivation1-PASS-c42.provn, and ordering-derivation1-PASS-c42.provx. Please note that, all though we provide three different representations, you only need to validate one per each test case.
For your convenience, the download links for all the available test cases (as enumerated in Section 2) for each representation are respectively provided in the following text files: rdf-tests.txt, provn-tests.txt, and xml-tests.txt.
Implementers are asked to validate each test case with their implementation and record the validation results in two lists: the pass list and the fail list. The pass list file MUST contain all the identifiers (one on each line) of the test cases that have been successfully validated, and the fail list MUST contain the identifiers of all those that have failed the validation. The identifiers MUST also include the file extension, e.g. ordering-derivation1-PASS-c42.provn, to indicate the versions that were actually tested. Test cases that are not supported by the implementation SHOULD NOT be included in either of the lists.
For example, if a validator can only process the PROV-N representations of the three test cases ordering-derivation1-PASS-c42, ordering-derivation2-FAIL-c42, and ordering-derivation3-PASS-c41-c42, we expect the result lists's contents to be similar to the below.
The pass list: ordering-derivation1-PASS-c42.provn ordering-derivation3-PASS-c41-c42.provn The fail list: ordering-derivation2-FAIL-c42.provn
Please report your test results using the constraint questionnaire.
This section enumerates all test cases available for PROV-CONSTRAINTS. There are two types of test cases:
All test cases SHOULD be checked for their validity with respect to the constraints they cover. A success test result for a test case means its validity MUST be the same as its expected validation result as provided in Table 2 below.
This section enumerates the example test cases from [[PROV-DM]].
This section enumerates the example test cases from [[PROV-O]].
Table 5 in this section provides coverage look up for the constraints in [[PROV-CONSTRAINTS]] against the available test cases.