--- a/data-cube-ucr/index.html Mon May 27 10:18:53 2013 +0200
+++ b/data-cube-ucr/index.html Tue May 28 13:35:08 2013 +0200
@@ -6,7 +6,7 @@
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
-<title>Use Cases and Requirements for the Data Cube Vocabulary</title>
+<title>Use Cases and Lessons for the Data Cube Vocabulary</title>
<script type="text/javascript"
src='../respec/respec3/builds/respec-w3c-common.js' class='remove'></script>
@@ -26,14 +26,17 @@
applications.</p>
<p>
In this document, the <a href="http://www.w3.org/2011/gld/">W3C
- Government Linked Data Working Group</a> presents use cases and
- requirements supporting a recommendation of the RDF Data Cube
- Vocabulary [<cite><a href="#ref-QB-2013">QB-2013</a></cite>]. The
- group obtained use cases from existing deployments of and experiences
+ Government Linked Data Working Group</a> presents use cases and lessons
+ supporting a recommendation of the RDF Data Cube Vocabulary [<cite><a
+ href="#ref-QB-2013">QB-2013</a></cite>]. The document describes use cases of
+ the data cube vocabulary from existing deployments of and experiences
with an earlier version of the data cube vocabulary [<cite><a
- href="#ref-QB-2010">QB-2010</a></cite>]. The group also describes a set of
- requirements derived from the use cases and to be considered in the
- recommendation.
+ href="#ref-QB-2010">QB-2010</a></cite>] as well as other possible use cases
+ that would benefit from using the vocabulary. In particular, the
+ document identifies benefits and challenges in using a vocabulary for
+ representing statistics. Also, it derives lessons that can be used for
+ future work on the vocabulary as well as for useful tools
+ complementing the vocabulary.
</p>
</section>
@@ -48,75 +51,73 @@
<section>
<h2 id="introduction">Introduction</h2>
- The aim of this document is to present concrete use cases and
- requirements for a vocabulary to publish statistics as Linked Data. An
- earlier version of the data cube vocabulary [<cite><a
- href="#ref-QB-2010">QB-2010</a></cite>] has existed for some time and
- has proven applicable in <a
- href="http://wiki.planet-data.eu/web/Datasets">several deployments</a>.
- The <a href="http://www.w3.org/2011/gld/">W3C Government Linked
- Data Working Group</a> intends to transform the data cube vocabulary into
- a W3C recommendation of the RDF Data Cube Vocabulary [<cite><a
- href="#ref-QB-2013">QB-2013</a></cite>]. This document describes use cases
- and requirements derived from existing data cube deployments in order
- to document and illustrate design decisions that have driven the work.
+ The aim of this document is to present concrete use cases and lessons
+ for a vocabulary to publish statistics as Linked Data. An earlier
+ version of the data cube vocabulary [<cite><a
+ href="#ref-QB-2010">QB-2010</a></cite>] has existed for some time and has
+ proven applicable in <a href="http://wiki.planet-data.eu/web/Datasets">several
+ deployments</a>. The <a href="http://www.w3.org/2011/gld/">W3C
+ Government Linked Data Working Group</a> intends to transform the data
+ cube vocabulary into a W3C recommendation of the RDF Data Cube
+ Vocabulary [<cite><a href="#ref-QB-2013">QB-2013</a></cite>]. The
+ document describes use cases that would benefit from using the
+ vocabulary. In particular, the document identifies possible benefits
+ and challenges in using such a vocabulary for representing statistics.
+ Also, it derives lessons that can motivate future work on the
+ vocabulary as well as associated tools or services complementing the
+ vocabulary.
<p>The rest of this document is structured as follows. We will
- first give a short introduction to modelling statistics. Then, we will describe use cases that have been derived
- from existing deployments or feedback to the earlier version of the data cube vocabulary. In particular, we describe possible benefits and
- challenges of use cases. Afterwards, we will describe concrete
- requirements that were derived from those use cases and that have been
- taken into account for the specification.</p>
+ first give a short introduction to modelling statistics. Then, we will
+ describe use cases that have been derived from existing deployments or
+ feedback to the earlier version of the data cube vocabulary. In
+ particular, we describe possible benefits and challenges of use cases.
+ Afterwards, we will describe concrete lessons that were derived from
+ those use cases.</p>
<p>We use the term "data cube vocabulary" throughout the document
when referring to the vocabulary.</p>
<section>
<h3 id="describingstatistics">Describing statistics</h3>
- <p>In the following, we describe the the challenge of authoring an RDF vocabulary
- for publishing statistics as Linked Data.</p>
+ <p>In the following, we describe the the challenge of authoring an
+ RDF vocabulary for publishing statistics as Linked Data.</p>
<p>Describing statistics - collected and aggregated numeric data -
is challenging for the following reasons:</p>
<ul>
<li>Representing statistics requires more complex modelling as
discussed by Martin Fowler [<cite><a href="#ref-FOWLER97">FOWLER97</a></cite>]:
Recording a statistic simply as an attribute to an object (e.g., the
- fact that a person weighs 185 pounds) fails to represent
- important concepts such as quantity, measurement, and unit. Instead,
- a statistic is modeled as a distinguishable object, an observation.
+ fact that a person weighs 185 pounds) fails to represent important
+ concepts such as quantity, measurement, and unit. Instead, a
+ statistic is modeled as a distinguishable object, an observation.
</li>
<li>The object describes an observation of a value, e.g., a
numeric value (e.g., 185) in case of a measurement or a categorical
value (e.g., "blood group A") in case of a categorical observation.</li>
- <li>To allow for correct interpretation of the value, the object can
- be further described by "dimensions", e.g., the specific phenomenon
- "weight" observed and the unit "pounds". Given background
- information, e.g., arithmetical and comparative operations, humans
- and machines can appropriately visualize such observations or have
- conversions between different quantities.</li>
- <li>Also, an observation separates a value from the actual event
- at which it was collected; for instance, one can describe the
- "Person" that collected the observation and the "Time" the
- observation was collected.</li>
+ <li>To allow correct interpretation of the value, the observation
+ needs to be further described by "dimensions" such as the specific
+ phenomenon, e.g., "weight", the time the observation is valid, e.g.,
+ "January 2013" or a location the observation was done, e.g., "New
+ York".</li>
+ <li>To further improve interpretation of the value, attributes
+ such as presentational information, e.g. a series title "COINS 2010
+ to 2013" or critical information to understanding the data, e.g. the
+ unit of measure "miles" can be given to observations.</li>
+ <li>Given background information, e.g., arithmetical and
+ comparative operations, humans and machines can appropriately
+ visualize such observations or have conversions between different
+ quantities.</li>
</ul>
- The following figure illustrates these details in a
- class diagram:
-
- <p class="caption">Figure: Modelling a statistic</p>
-
- <p align="center">
- <img alt="specificity of modelling a
- statistic"
- src="./figures/modeling_quantity_measurement_observation.png"></img>
- </p>
<p>
The Statistical Data and Metadata eXchange [<cite><a
href="#ref-SDMX">SDMX</a></cite>] - the ISO standard for exchanging and
- sharing statistical data and metadata among organizations -
- uses a "multidimensional model" to meet the above challenges in modelling statistics. It can describe statistics as observations.
- Observations exhibit values (Measures) that depend on dimensions
- (Members of Dimensions).
+ sharing statistical data and metadata among organizations - uses a
+ "multidimensional model" to meet the above challenges in modelling
+ statistics. It can describe statistics as observations. Observations
+ exhibit values (Measures) that depend on dimensions (Members of
+ Dimensions).
</p>
<p>Since the SDMX standard has proven applicable in many contexts,
the vocabulary adopts the multidimensional model that underlies SDMX
@@ -139,13 +140,14 @@
<dfn>statistical data</dfn>
is a multidimensional table (also called a data cube) [<cite><a
href="#ref-SDMX">SDMX</a></cite>], i.e., a set of observed values organized
- along a group of dimensions, together with associated metadata. We refer to aggregated statistical
- data as "macro-data" and unaggregated statistical data as "micro-data".
+ along a group of dimensions, together with associated metadata. We
+ refer to aggregated statistical data as "macro-data" and unaggregated
+ statistical data as "micro-data".
</p>
<p>
Statistical data can be collected in a
- <dfn>dataset</dfn>
- , typically published and maintained by an organization [<cite><a
+ <dfn>dataset,</dfn>
+ typically published and maintained by an organization [<cite><a
href="#ref-SDMX">SDMX</a></cite>]. The dataset contains metadata, e.g.,
about the time of collection and publication or about the maintaining
and publishing organization.
@@ -153,8 +155,8 @@
<p>
<dfn>Source data</dfn>
- is data from datastores such as relational databases or spreadsheets that acts as a
- source for the Linked Data publishing process.
+ is data from datastores such as relational databases or spreadsheets
+ that acts as a source for the Linked Data publishing process.
</p>
<p>
@@ -185,7 +187,9 @@
<p>
A
<dfn>registry</dfn>
- allows a publisher to announce that data or metadata exists and to add information about how to obtain that data. [<cite><a href="#ref-SDMX-21">SDMX 2.1</a></cite>]
+ allows a publisher to announce that data or metadata exists and to add
+ information about how to obtain that data. [<cite><a
+ href="#ref-SDMX-21">SDMX 2.1</a></cite>]
</p>
</section>
@@ -218,21 +222,8 @@
consumption application (consumer) that first discovers data from the
registry, then queries data from the corresponding publisher of
selected data, and then visualises the data.</p>
- <p>In the following, we illustrate the processes from this use case
- in a flow diagram by SDMX and describe what activities are enabled in
- this use case by having statistics described in a machine-readable
- format.</p>
- <p class="caption">
- Figure: Process flow diagram by SDMX [<cite><a
- href="#ref-SDMX-21">SDMX 2.1</a></cite>]
- </p>
-
- <p align="center">
- <img alt="SDMX Web Dissemination Use Case"
- src="./figures/SDMX_Web_Dissemination_Use_Case.png" width="1000px"></img>
- </p>
- <p>Benefits:</p>
+ <h4>Benefits</h4>
<ul>
<li>A structural metadata source (registry) can collect metadata
about statistical data.</li>
@@ -240,34 +231,37 @@
<li>A data web service (publisher) can register statistical data
in a registry, and can provide statistical data from a database and
metadata from a metadata repository for consumers. For that, the
- publisher creates database tables (see 1 in figure), and loads
- statistical data in a database and metadata in a metadata repository.</li>
+ publisher creates database tables, and loads statistical data in a
+ database and metadata in a metadata repository.</li>
- <li>A consumer can discover data from a registry (3) and
+ <li>A consumer can discover data from a registry and
automatically can create a query to the publisher for selected
- statistical data (4).</li>
+ statistical data.</li>
<li>The publisher can translate the query to a query to its
- database (5) as well as metadata repository (6) and return the
- statistical data and metadata.</li>
+ database as well as metadata repository and return the statistical
+ data and metadata.</li>
<li>The consumer can visualise the returned statistical data and
metadata.</li>
</ul>
- <p>Requirements:</p>
+ <h4>Challenges</h4>
<ul>
- <li><a
- href="#Thereshouldbearecommendedwaytocommunicatetheavailabilityofpublishedstatisticaldatatoexternalpartiesandtoallowautomaticdiscoveryofstatisticaldata">There
- should be a recommended way to communicate the availability of
- published statistical data to external parties and to allow
- automatic discovery of statistical data</a></li>
+ <li>This use case is too abstract. The SDMX Web Dissemination Use
+ Case can be concretised by several sub-use cases, detailed in the
+ following sections.</li>
+ <li>In particular, this use case requires a recommended way to
+ advertise published statistical datasets, which supports the
+ following lesson: <a
+ href="#Thereshouldbearecommendedwaytocommunicatetheavailabilityofpublishedstatisticaldatatoexternalpartiesandtoallowautomaticdiscoveryofstatisticaldata">Publishers
+ may need guidance in communicating the availability of published
+ statistical data to external parties and to allow automatic
+ discovery of statistical data</a>
+ </li>
+
</ul>
-
- <p>The SDMX Web Dissemination Use Case can be concretised by
- several sub-use cases, detailed in the following sections.</p>
-
</section> <section>
<h3 id="UKgovernmentfinancialdatafromCombinedOnlineInformationSystem">Publisher
Use Case: UK government financial data from Combined Online
@@ -314,44 +308,60 @@
Treasury is updated daily. In principle at least, multiple snapshots
of the COINS data could be released through the year.</p>
+ <p>The actual data and its hypercube structure are to be
+ represented separately so that an application first can examine the
+ structure before deciding to download the actual data, i.e., the
+ transactions. The hypercube structure also defines for each dimension
+ and attribute a range of permitted values that are to be represented.</p>
+ <p>An access or query interface to the COINS data, e.g., via a
+ SPARQL endpoint or the linked data API, is planned. Queries that are
+ expected to be interesting are: "spending for one department", "total
+ spending by department", "retrieving all data for a given observation"
+ etc.</p>
+
+ <h4>Benefits</h4>
<p>According to the COINS as Linked Data project, the reason for
publishing COINS as Linked Data are threefold:</p>
<ul>
- <li>using open standard representation makes it easier to work
+ <li>Using open standard representation makes it easier to work
with the data with available technologies and promises innovative
third-party tools and usages</li>
- <li>individual transactions and groups of transactions are given
+ <li>Individual transactions and groups of transactions are given
an identity, and so can be referenced by web address (URL), to allow
them to be discussed, annotated, or listed as source data for
articles or visualizations</li>
- <li>cross-links between linked-data datasets allow for much
+ <li>Cross-links between linked-data datasets allow for much
richer exploration of related datasets</li>
</ul>
+ <h4>Challenges</h4>
+
<p>The COINS use case leads to the following challenges:</p>
<ul>
- <li>The actual data and its hypercube structure are to be
- represented separately so that an application first can examine the
- structure before deciding to download the actual data, i.e., the
- transactions. The hypercube structure also defines for each dimension
- and attribute a range of permitted values that are to be represented.</li>
- <li>An access or query interface to the COINS data, e.g., via a
- SPARQL endpoint or the linked data API, is planned. Queries that are
- expected to be interesting are: "spending for one department", "total
- spending by department", "retrieving all data for a given
- observation",</li>
+ <li>Although not originally not intended, the data cube
+ vocabulary could be successfully used for publishing financial data,
+ not just statistics.</li>
<li>Also, the publisher favours a representation that is both as
self-descriptive as possible, i.e., others can link to and download
fully-described individual transactions and as compact as possible,
- i.e., information is not unnecessarily repeated.</li>
+ i.e., information is not unnecessarily repeated. This challenge
+ supports lesson: <a
+ href="#Vocabularyshouldclarifytheuseofsubsetsofobservations">Publishers
+ may need more guidance in creating and managing slices or arbitrary
+ groups of observations</a>
+ </li>
<li>Moreover, the publisher is thinking about the possible
benefit of publishing slices of the data, e.g., datasets that fix all
dimensions but the time dimension. For instance, such slices could be
particularly interesting for visualisations or comments. However,
depending on the number of Dimensions, the number of possible slices
can become large which makes it difficult to select all interesting
- slices.</li>
+ slices. This challenge supports lesson: <a
+ href="#Vocabularyshouldclarifytheuseofsubsetsofobservations">Publishers
+ may need more guidance in creating and managing slices or arbitrary
+ groups of observations</a>
+ </li>
<li>An important benefit of linked data is that we are able to
annotate data, at a fine-grained level of detail, to record
information about the data itself. This includes where it came from –
@@ -359,16 +369,18 @@
reviewers, links to other useful resources, etc. Being able to trust
that data to be correct and reliable is a central value for
government-published data, so recording provenance is a key
- requirement for the COINS data.</li>
+ requirement for the COINS data. This challenge supports lesson: <a
+ href="#Vocabularyshouldclarifytheuseofsubsetsofobservations">Publishers
+ may need more guidance in creating and managing slices or arbitrary
+ groups of observations</a>
+ </li>
<li>A challenge also is the size of the data, especially since it
is updated regularly. Five data files already contain between 3.3 and
- 4.9 million rows of data.</li>
- </ul>
- <p>Requirements::</p>
- <ul>
- <li><a
- href="#Vocabularyshouldclarifytheuseofsubsetsofobservations">Vocabulary
- should clarify the use of subsets of observations</a></li>
+ 4.9 million rows of data. This challenge supports lesson: <a
+ href="#Vocabularyshouldclarifytheuseofsubsetsofobservations">Publishers
+ may need more guidance in creating and managing slices or arbitrary
+ groups of observations</a>
+ </li>
</ul>
</section> <section>
@@ -486,23 +498,23 @@
<p>Example (in pseudo-turtle RDF):</p>
<pre>
ex:obs1
- sdmx:refArea <uk>;
+ sdmx:refArea <uk>;
sdmx:refPeriod "2011";
ex:population "60" .
ex:obs2
- sdmx:refArea <england>;
+ sdmx:refArea <england>;
sdmx:refPeriod "2011";
ex:population "50" .
ex:obs3
- sdmx:refArea <scotland>;
+ sdmx:refArea <scotland>;
sdmx:refPeriod "2011";
ex:population "5" .
ex:obs4
- sdmx:refArea <wales>;
+ sdmx:refArea <wales>;
sdmx:refPeriod "2011";
ex:population "3" .
ex:obs5
- sdmx:refArea <northernireland>;
+ sdmx:refArea <northernireland>;
sdmx:refPeriod "2011";
ex:population "2" .
@@ -560,7 +572,8 @@
</section> <section>
<h3 id="PublishingslicesofdataaboutUKBathingWaterQuality">Publisher
- Use Case: Publishing slices of data about UK Bathing Water Quality</h3>
+ Use Case: Publishing Observational Data Sets about UK Bathing Water
+ Quality</h3>
<p>
<span style="font-size: 10pt">(Use case has been provided by
Epimorphics Ltd, in their <a
@@ -1219,16 +1232,16 @@
</section> </section>
<section>
- <h2 id="requirements">Requirements</h2>
+ <h2 id="requirements">Lessons</h2>
<p>The use cases presented in the previous section give rise to the
- following requirements for a standard representation of statistics.
- Requirements are cross-linked with the use cases that motivate them.</p>
-
+ following lessons that can motivate future work on the vocabulary as
+ well as associated tools or services complementing the vocabulary.</p>
<section>
- <h3 id="VocabularyshouldbuildupontheSDMXinformationmodel">Vocabulary
- should build upon the SDMX information model</h3>
+ <h3 id="VocabularyshouldbuildupontheSDMXinformationmodel">There is
+ a putative requirement to update to SDMX 2.1 if there are specific use
+ cases that demand it</h3>
<p>
The draft version of the vocabulary builds upon <a
href="http://sdmx.org/?page_id=16">SDMX Standards Version 2.0</a>. A
@@ -1256,8 +1269,9 @@
</ul>
</section> <section>
- <h3 id="Vocabularyshouldclarifytheuseofsubsetsofobservations">Vocabulary
- should clarify the use of subsets of observations</h3>
+ <h3 id="Vocabularyshouldclarifytheuseofsubsetsofobservations">Publishers
+ may need more guidance in creating and managing slices or arbitrary
+ groups of observations</h3>
<p>There should be a consensus on the issue of flattening or
abbreviating data; one suggestion is to author data without the
duplication, but have the data publication tools "flatten" the compact
@@ -1285,8 +1299,9 @@
</section> <section>
<h3
- id="Vocabularyshouldrecommendamechanismtosupporthierarchicalcodelists">Vocabulary
- should recommend a mechanism to support hierarchical code lists</h3>
+ id="Vocabularyshouldrecommendamechanismtosupporthierarchicalcodelists">Publishers
+ may need more guidance to decide which representation of hierarchies
+ is most suitable for their use case</h3>
<p>
First, hierarchical code lists may be supported via SKOS [<cite><a
href="#ref-skos">SKOS</a></cite>]. Allow for cross-location and cross-time
@@ -1331,8 +1346,9 @@
</section> <section>
<h3
- id="VocabularyshoulddefinerelationshiptoISO19156ObservationsMeasurements">Vocabulary
- should define relationship to ISO19156 - Observations & Measurements</h3>
+ id="VocabularyshoulddefinerelationshiptoISO19156ObservationsMeasurements">Modelers
+ using ISO19156 - Observations & Measurements may need clarification
+ regarding the relationship to the data cube vocabulary</h3>
<p>An number of organizations, particularly in the Climate and
Meteorological area already have some commitment to the OGC
"Observations and Measurements" (O&M) logical data model, also
@@ -1352,9 +1368,9 @@
</section> <section>
<h3
- id="Thereshouldbearecommendedmechanismtoallowforpublicationofaggregateswhichcrossmultipledimensions">There
- should be a recommended mechanism to allow for publication of
- aggregates which cross multiple dimensions</h3>
+ id="Thereshouldbearecommendedmechanismtoallowforpublicationofaggregateswhichcrossmultipledimensions">Publishers
+ may need guidance in how to represent common analytical operations
+ such as Slice, Dice, Rollup on data cubes</h3>
<p>Background information:</p>
<ul>
@@ -1374,8 +1390,9 @@
</ul>
</section> <section>
- <h3 id="Thereshouldbearecommendedwayofdeclaringrelationsbetweencubes">There
- should be a recommended way of declaring relations between cubes</h3>
+ <h3 id="Thereshouldbearecommendedwayofdeclaringrelationsbetweencubes">Publishers
+ may need guidance in making transparent the pre-processing of
+ aggregate statistics</h3>
<p>Background information:</p>
<ul>
<li>Issue: <a href="http://www.w3.org/2011/gld/track/issues/30">http://www.w3.org/2011/gld/track/issues/30</a></li>
@@ -1393,9 +1410,9 @@
</section> <section>
<h3
- id="Thereshouldbecriteriaforwell-formednessandassumptionsconsumerscanmakeaboutpublisheddata">There
- should be criteria for well-formedness and assumptions consumers can
- make about published data</h3>
+ id="Thereshouldbecriteriaforwell-formednessandassumptionsconsumerscanmakeaboutpublisheddata">Publishers
+ and consumers may need guidance in checking and making use of
+ well-formedness of published data using data cube</h3>
<p>Background information:</p>
<ul>
@@ -1420,9 +1437,9 @@
</section> <section>
<h3
- id="Thereshouldbemechanismsandrecommendationsregardingpublicationandconsumptionoflargeamountsofstatisticaldata">There
- should be mechanisms and recommendations regarding publication and
- consumption of large amounts of statistical data</h3>
+ id="Thereshouldbemechanismsandrecommendationsregardingpublicationandconsumptionoflargeamountsofstatisticaldata">Publishers
+ and consumers may need more guidance in efficiently processing data
+ using the data cube vocabulary</h3>
<p>Background information:</p>
<ul>
<li>Related issue regarding abbreviations <a
@@ -1438,10 +1455,10 @@
</section> <section>
<h3
- id="Thereshouldbearecommendedwaytocommunicatetheavailabilityofpublishedstatisticaldatatoexternalpartiesandtoallowautomaticdiscoveryofstatisticaldata">There
- should be a recommended way to communicate the availability of
- published statistical data to external parties and to allow automatic
- discovery of statistical data</h3>
+ id="Thereshouldbearecommendedwaytocommunicatetheavailabilityofpublishedstatisticaldatatoexternalpartiesandtoallowautomaticdiscoveryofstatisticaldata">Publishers
+ may need guidance in communicating the availability of published
+ statistical data to external parties and to allow automatic discovery
+ of statistical data</h3>
<p>Clarify the relationship between DCAT and QB.</p>
<p>Background information:</p>
<ul>
@@ -1459,9 +1476,9 @@
</section> </section>
<section class="appendix">
<h2 id="acknowledgements">Acknowledgements</h2>
- <p>We thank John Erickson, Rinke Hoekstra, Bernadette Hyland, Aftab
- Iqbal, Dave Reynolds, Biplav Srivastava, Villazón-Terrazas for
- feedback and input.</p>
+ <p>We thank Phil Archer, John Erickson, Rinke Hoekstra, Bernadette
+ Hyland, Aftab Iqbal, James McKinney, Dave Reynolds, Biplav Srivastava,
+ Boris Villazón-Terrazas for feedback and input.</p>
</section>
<h2 id="references">References</h2>