From LOD to LOUD:
making data usable

Workshop at SWIB18

Fabian Steeg / @fsteeg & Adrian Pohl / @acka47 & Pascal Christoph / @dr0ide
Linked Open Data, Hochschulbibliothekszentrum NRW (hbz)

Bonn, 2018-11-26

This presentation:
http://hbz.github.io/swib18-workshop

Overview

Our background & experience

Hochschulbibliothekszentrum des Landes Nordrhein-Westfalen, est. 1973

= Academic library center of North Rhine-Westphalia

Software services for libraries in NRW and beyond

E.g. union catalog, discovery portal DigiBib, ILL, digitization & digital preservation, consortial acquisition

SWIB co-organizer

hbz flyer in English (PDF)

lobid: hbz's LOD service, providing bibliographic data, authorities & organizations

Started 2010 with open data publication, triple store & Perl transformation script

Original team: Adrian, Pascal, Felix. Fabian joined in 2012.

2013: First API release based on JSON-LD & Elasticsearch

2017–2018: LOUD with lobid API 2.0

Your background & experience

LOUD

Linked Open Usable Data (LOUD)

Source: Rob Sanderson, "Shout it Out: LOUD", CC-BY (video)

Source: Rob Sanderson, "Shout it Out: LOUD", CC-BY (video)

Design for JSON-LD
from RDF, consistently

Part I:
Convert RDF data into usable JSON-LD

Part Ia.

Usable data & APIs

APIs: why, how

Usable data [& APIs]

Libraries are Software

Our collections and services are delivered primarily via software. [...] The choices we make in the development, selection, and implementation of this software [...] define the limits of our content and services. We can only be as good as our software.

— Cody Hanson, Libraries are Software

[Usable data &] APIs

e.g. lobid sources, formats, applications

API: why?

API: why?

API: how?

API: how?

GET https://api.github.com

{
  "current_user_url": "https://api.github.com/user",
  "authorizations_url": "https://api.github.com/authorizations",
  "emails_url": "https://api.github.com/user/emails",
  "emojis_url": "https://api.github.com/emojis",
  "events_url": "https://api.github.com/events",
  "feeds_url": "https://api.github.com/feeds",
  "followers_url": "https://api.github.com/user/followers",
  "gists_url": "https://api.github.com/gists{/gist_id}",
  "hub_url": "https://api.github.com/hub",
  ...
}
					

Part Ib.

JSON APIs

Using the GitHub API, processing JSON

Setup

cURL

Command line tool for transferring data with URLs

https://curl.haxx.se/download.html

$ curl --help

Exercise 1a: cURL

$ curl https://api.github.com
											

# Copy above, paste in terminal
$ curl https://api.github.com
											

$ ↑ # access the history
$ curl https://api.github.com
											

(You can copy and paste in the terminal with Shift+Ctrl+C and Shift+Ctrl+V)

Exercise 1b: cURL

$ curl https://api.github.com/<???> # user or org
											

$ curl https://api.github.com/<???>
											

('<???>' means you should insert your solution here)

We have a repo for this workshop called 'swib18-workshop' under the 'hbz' organization

Exercise 1c: cURL
Solution

$ curl https://api.github.com/users/hbz
											

$ curl https://api.github.com/repos/hbz/swib18-workshop
											

('$' means you should type this in your terminal application)

Response

$ curl https://api.github.com/repos/hbz/swib18-workshop
{
  "id": 150073510,
  "node_id": "MDEwOlJlcG9zaXRvcnkxNTAwNzM1MTA=",
  "name": "swib18-workshop",
  "full_name": "hbz/swib18-workshop",
  "private": false,
  "owner": {
    "login": "hbz",
    "id": 6557108,
    "node_id": "MDEyOk9yZ2FuaXphdGlvbjY1NTcxMDg=",
    "avatar_url": "https://avatars3.githubusercontent.com/u/6557108?v=4",
    "gravatar_id": "",
						

Output is quite long, we often want specific values

JavaScript Object Notation (JSON)

var options = {
  url: 'https://api.github.com/repos/hbz/swib18-workshop',
  headers: { 'User-Agent': 'hbz' }
};

request(options, function (error, response, body) {
  var doc = JSON.parse(body);
  console.log('repo license:', doc.license.name) // <-- use JSON
});
						

> license.name: Creative Commons Attribution 4.0 International
						

jq

A lightweight and flexible command-line JSON processor

https://stedolan.github.io/jq/

https://jqplay.org/

$ jq --help

Exercise 2a: JSON with jq

$ curl https://api.github.com/repos/hbz/swib18-workshop \
| jq .name # filter: .name
						

('$' means you should type this in your terminal application)

Exercise 2b: JSON with jq

$ curl https://api.github.com/repos/hbz/swib18-workshop | jq <???> # get the number of open issues
										

$ curl https://api.github.com/repos/hbz/swib18-workshop | jq <???> # get the repo's license name
										

('<???>' means you should insert your solution here)

Exercise 2c: JSON with jq

$ curl https://api.github.com/repos/hbz/swib18-workshop | jq .open_issues_count
										

$ curl https://api.github.com/repos/hbz/swib18-workshop | jq .license.name
										

('$' means you should type this in your terminal application)

So JSON is great.

Part Ic.

From RDF to JSON

JSON-LD processing with jsonld.js via jsonld-cli

Bibframe datasets (N-Triples)

Example work

JSON-LD

5. Design for JSON-LD, consistently

Great, so let's make JSON-LD from RDF

jsonld-cli

JSON-LD command line interface tool

https://github.com/hbz/jsonld-cli

Our fork of https://github.com/digitalbazaar/jsonld-cli, added import from N-Quads

$ jsonld --help

Exercise 3:
N-Triples to JSON-LD

$ cd ~/git/swib18-workshop/data
										

$ jsonld import loc.nt
										

$ jsonld import loc.nt > loc.json
										

('$' means you should type this in your terminal application)

Result

{
 "@id": "http://id.loc.gov/resources/works/c000101650",
 "http://id.loc.gov/ontologies/bibframe/subject": [{
  "@id": "http://id.loc.gov/resources/works/101650#Topic650-20"
 }, ...]
}, {
 "@id": "http://id.loc.gov/resources/works/101650#Topic650-20",
 "@type": [
  "http://id.loc.gov/ontologies/bibframe/Topic",
  "http://www.loc.gov/mads/rdf/v1#ComplexSubject"
 ],
 "http://www.w3.org/2000/01/rdf-schema#label": [{
  "@value": "Climatic changes--Europe."
 }],...
}
					

Serialized RDF as JSON-LD | Specification

Exercise 3a:
Use serialized RDF to access subjects

$ cat loc.json | less
										

# like .license.name, but for subject labels?
$ cat loc.json | jq <???>
										

So JSON is great?

Result

{
 "@id": "http://id.loc.gov/resources/works/c000101650",
 "http://id.loc.gov/ontologies/bibframe/subject": [{
  "@id": "http://id.loc.gov/resources/works/101650#Topic650-20"
 }, ...]
}, {
 "@id": "http://id.loc.gov/resources/works/101650#Topic650-20",
 "@type": [
  "http://id.loc.gov/ontologies/bibframe/Topic",
  "http://www.loc.gov/mads/rdf/v1#ComplexSubject"
 ],
 "http://www.w3.org/2000/01/rdf-schema#label": [{
  "@value": "Climatic changes--Europe."
 }],...
}
					

Serialized RDF as JSON-LD | Specification

Data processing vs. API access

for o in doc :
		if o["@type"] contains "http://id.loc.gov/ontologies/bibframe/Topic" :
			subjectId = o["http://id.loc.gov/ontologies/bibframe/subject"][0]["@id"] // + error handling

for o in doc :
	if o["@id"] == subjectId :
		subjectLabel = o["http://www.w3.org/2000/01/rdf-schema#label"]["@value"] // + error handling
										

doc.license.name // doc.subject.label?
										

Developer empathy

Non-developer empathy

★ Few barriers to entry

Design for JSON-LD from RDF, consistently

Framing

Exercise 4:
Framing

{
 "@type": "http://id.loc.gov/ontologies/bibframe/Work",
 "@embed": "@always"
}
										

$ jsonld frame -f frame.json loc.json
										

$ jsonld frame -f frame.json loc.json > loc-framed.json
										

('$' means you should type this in your terminal application)

Result

{
 "@id": "http://id.loc.gov/resources/works/c000101650",
 "http://id.loc.gov/ontologies/bibframe/subject": [{
   "@id":"http://id.loc.gov/resources/works/101650#Topic650-20",
   "@type": [
     "http://id.loc.gov/ontologies/bibframe/Topic",
     "http://www.loc.gov/mads/rdf/v1#ComplexSubject"
   ],
   ...
   "http://www.w3.org/2000/01/rdf-schema#label": "Climatic changes--Europe."
 }, {...}],
}
					

Framed JSON-LD | Specification

Exercise 4a:
Use framed JSON-LD to access subjects

$ cat loc-framed.json | less
										

# like .license.name, but for subject labels?
$ cat loc-framed.json | jq <???>
										

2. As simple as possible?

JSON-LD Context

How do we get the context?

Context: more than simple mappings

Consistency

Describing values

"contribution": {
  "@id": "http://id.loc.gov/ontologies/bibframe/contribution",
  "@container": "@set"
}
						

Exercise 5:
Compact

$ jsonld compact -c context.json loc-framed.json
										

$ jsonld compact -c context.json loc-framed.json > loc-compact.json
										

('$' means you should type this in your terminal application)

Result

{
 "@id": "http://id.loc.gov/resources/works/c000101650",
 "subject": [
  {
   "id": "http://id.loc.gov/resources/works/101650#Topic650-20",
   "type": [
    "Topic",
    "ComplexSubject"
   ],
   "label": "Climatic changes--Europe."
  }
 ], ...
}
					

Compacted JSON-LD | Specification

Exercise 5a:
Use compact JSON-LD to access subjects

$ cat loc-compact.json | less
										

$ cat loc-compact.json | jq .label
										

# like .license.name, but for subject labels?
$ cat loc-compact.json | jq <???>
										

('$' means you should type this in your terminal application)

Exercise 5a:
Solution

$ cat loc-compact.json | jq .subject.label

jq: error: Cannot index array with string "label"
										

$ cat loc-compact.json | jq .subject[0].label
										

$ cat loc-compact.json | jq .subject[].label
										

('$' means you should type this in your terminal application)

Missing values in context

Exercise 5b: Fix context

$ cat loc-compact.json | jq -c 'path(..)'
										

$ cat loc-compact.json | jq -c 'path(..)' | grep http
										

"value" : "http://www.w3.org/1999/02/22-rdf-syntax-ns#value"
										

$ jsonld compact -c context.json loc-framed.json > loc-compact.json
										

('$' means you should type this in your terminal application)

Exercise 6: Use final JSON

$ cat loc-compact.json | less
										

$ cat loc-compact.json | jq <???> # get authors
										

$ cat loc-compact.json | jq <???> # get subtitle
										

$ jsonld format -q data/loc-compact.json
										

('$' means you should type this in your terminal application)

Exercise 6: Solutions

$ cat loc-compact.json | jq .contribution[].agent.label
										

$ cat loc-compact.json | jq .title.subtitle
										

('$' means you should type this in your terminal application)

So JSON is great

RDF view

1. Require use cases, with data

JSON view

Context maps stable API to implementation details

JSON with context

(based on part of https://api.github.com/users/hbz)

{
  "@id": "https://github.com/users/hbz",
  "type": "Organization",
  "name": "hbz",
  "blog": "https://www.hbz-nrw.de",

  "@context": {
    "type": "@type",
    "name": "https://schema.org/name",
    "blog": { "@id": "https://schema.org/url", "@type": "@id" },
    "Organization": "https://schema.org/Organization"
  }
}
					

You can test this in the JSON-LD Playground

JSON as RDF

{
  "@context": "...",
  "type": "Organization",
  "name": "hbz",
  "blog": "https://www.hbz-nrw.de",
}
					

<https://github.com/users/hbz> \
<http://www.w3.org/1999/02/22-rdf-syntax-ns#type> \
<https://schema.org/Organization> .
<https://github.com/users/hbz> <https://schema.org/name> "hbz" .
<https://github.com/users/hbz> <https://schema.org/url> \
<https://www.hbz-nrw.de> .
					

Output of N-Quads tab in the JSON-LD Playground

Part I: Convert RDF data into usable JSON-LD

Part II:
Index and access the data with Elasticsearch

Part IIa.

Elasticsearch bulk indexing

Create bulk format with jq, index with curl

Elasticsearch

Exercise 7: Convert more data (100 documents)

$ tail loc-100.nt
										

$ jsonld import <???>
										

$ jsonld frame <???>
										

$ jsonld compact <???>
										

('$' means you should type this in your terminal application)

Exercise 7: Solutions

$ jsonld import loc-100.nt > loc-100.json
										

$ jsonld frame -f frame.json loc-100.json > loc-100-framed.json
										

$ jsonld compact -c context.json loc-100-framed.json > loc-100-compact.json
										

('$' means you should type this in your terminal application)

Elasticsearch bulk format

{"index":{"_index":"loc","_type":"work","_id":"c000101650"}}

{@context: ...,"id":"http://...","type":["Work","Text"],...}

Input data format

{
  "@context": { ... },
  "@graph": [
    { "id": "http://.../1", "type": ["Work", "Text"], ... },
    { "id": "http://.../2", "type": ["Work", "Text"], ... },
    ...
  ]
}
							

Exercise 8a: Document line

$ cat loc-100-compact.json | jq
										

$ cat loc-100-compact.json | jq -c .
										

$ cat loc-100-compact.json | jq -c '.["@graph"][]'
										

@graph requires special escaping in jq with ', [], "

('$' means you should type this in your terminal application)

Exercise 8b: Metadata line

$ cat loc-100-compact.json | \
jq -c '.["@graph"][] | .'
									

$ cat loc-100-compact.json | \
jq -c '.["@graph"][] | { test:1 }'
										

$ cat loc-100-compact.json | \
jq -c '.["@graph"][] | { _id:.id }'
										

$ cat loc-100-compact.json | \
jq -c '.["@graph"][] | { "index":{"_index":"loc", "_type":"work", "_id":.id} }'
										

Exercise 8c: Putting it together

$ cat loc-100-compact.json | jq -r \
'.["@graph"][] | "\({meta:1})\n\({data:2})"'
										

$ cat loc-100-compact.json | jq -r \
'.["@graph"][] | "\({ "index":{"_index":"loc", "_type":"work", "_id":.id} })\n\(.)"' \
> bulk.ndjson
										

('$' means you should type this in your terminal application)

Elasticsearch bulk API

Setup

You Know,
for Search

http://localhost:9200

Exercise 9: Bulk indexing

$ tail -n 2 bulk.ndjson
										

$ curl -H "Content-Type: application/x-ndjson" \
-XPOST localhost:9200/_bulk \
--data-binary "@bulk.ndjson" | jq
										

$ curl localhost:9200/loc/work/_search | jq
										

('$' means you should type this in your terminal application)

Part IIb.

Access the Elasticsearch index

Search queries, direct access by ID

Elasticsearch URI search

Exercise 10a: Query syntax

$ curl localhost:9200/loc/work/_search?q=<???> | jq
										

# <field>:<term>
$ curl localhost:9200/loc/work/_search?q=<???> | jq
										

# AND, OR
$ curl localhost:9200/loc/work/_search?q=<???> | jq
										

# truncate with *
$ curl localhost:9200/loc/work/_search?q=<???> | jq
										

('<???>' means you should insert your solution here)

Exercise 10b: Solutions

$ curl localhost:9200/loc/work/_search?q=america | jq
										

$ curl localhost:9200/loc/work/_search?q=label:letters | jq
										

$ curl localhost:9200/loc/work/_search?q=canada+OR+alaska | jq
										

$ curl localhost:9200/loc/work/_search?q=ameri* | jq
										

('$' means you should type this in your terminal application)

Direct access by ID

Exercise 11a: Re-index

$ cat loc-compact.json | jq '(.id / "/") | last'
										

$ cat loc-100-compact.json | jq -r \
'.["@graph"][] | "\({ "index":{"_index":"loc", "_type":"work", "_id":(.id / "/") | last} })\n\(.)"' > bulk.ndjson
										

$ curl -XDELETE localhost:9200/loc | jq
										

$ curl -H "Content-Type: application/x-ndjson" \
-XPOST localhost:9200/_bulk \
--data-binary "@bulk.ndjson" | jq
										

Exercise 11b: GET by ID

$ curl "localhost:9200/<???>/<???>/<???>" \
| jq
										

$ curl \
"localhost:9200/<???>/<???>/<???>/_source" \
| jq
										

('<???>' means you should insert your solution here)

Exercise 11c: Solutions

$ curl "localhost:9200/loc/work/c000000026" \
| jq
										

$ curl "localhost:9200/loc/work/c000000026/_source" \
| jq
										

('$' means you should type this in your terminal application)

Direct access by ID

Exercise 12a: Remote JSON to N-Quads

$ jsonld format -q loc-compact.json
										

$ jsonld format -q http://localhost:9200/loc/work/c000000026/_source
										

Input data format

{
  "@context": { ... },
  "@graph": [
    { "id": "http://.../1", "type": ["Work", "Text"], ... },
    { "id": "http://.../2", "type": ["Work", "Text"], ... },
    ...
  ]
}
						

We processed & indexed each element of the @graph array

Add context

Exercise 12b: Serve context

$ cd ~/git/swib18-workshop ; npm start
										

$ curl http://localhost:3000/context.json | jq
										

('$' means you should type this in your terminal application)

Exercise 12c: Link to context

$ cat loc-100-compact.json | \
jq -c '.["@graph"][] | . + {"test": 1}'
										

$ cat loc-100-compact.json | \
jq -c '.["@graph"][] | . + {"@context":"http://localhost:3000/context.json"}'
										

$ cat loc-100-compact.json | jq -r \
'.["@graph"][] | "\({ "index":{"_index":"loc", "_type":"work", "_id":(.id / "/") | last} })\n\({"@context":"http://localhost:3000/context.json"}+.)"' > bulk.ndjson
										

Exercise 12d: Reindex and use

$ curl -XDELETE localhost:9200/loc | jq
										

$ curl -H "Content-Type: application/x-ndjson" \
-XPOST localhost:9200/_bulk \
--data-binary "@bulk.ndjson" | jq
										

$ jsonld format -q http://localhost:9200/loc/work/c000000026/_source
										

('$' means you should type this in your terminal application)

API: How — JSON over HTTP

Serving our data as JSON over HTTP, serializable as RDF

Elasticsearch: so much more

Boosting, Analyzers, Aggregations, etc.

Elasticsearch: The Definitive Guide (concepts)

Elasticsearch Reference (current API)

Part II: Index and access the data with Elasticsearch

Part III:
Use the data to build a web application

Part IIIa.

Search page

HTML page for index search

Exercise 13: HTML

$ gedit search.html &
										

<!DOCTYPE html>
<html>
	<head>
		<meta charset="UTF-8">
		<title>Title</title>
	</head>
	<body>
		Hello HTML
	</body>
</html>
										

$ chromium-browser search.html &
										

JQuery

Exercise 14: Text Input

<head>
  <!-- previous head content -->
  <script src="https://code.jquery.com/jquery-3.3.1.min.js">
  </script>
</head>
						

<body>
	<input id='search' type='text'/>
	<script>
		$('#search').keyup(function() {
			var q = $('#search').val();
			console.log(q);
		});
	</script>
	<div id='result'></div>
</body>
						

(Open your browser's developer tools with Ctrl+Shift+I to see the console.log output)

AJAX

Exercise 15: AJAX request

$('#search').keyup(function() { // <-- insert after this
  var q = encodeURIComponent($('#search').val());
  $.ajax({
	url: `http://localhost:9200/loc/work/_search?&q=${q}`,
	success: function(result) {
		console.log(JSON.stringify(result));
	},
	error: function(jqXHR, textStatus, errorThrown) {
		console.log(textStatus, ': ', errorThrown);
	}
  });
});
						

(Open your browser's developer tools with Ctrl+Shift+I to see the console.log output)

Cross-Origin Resource Sharing (CORS)

Exercise 16: Configure CORS

$ sudo gedit /etc/elasticsearch/elasticsearch.yml
										

"http": {
    "cors.enabled": true,
    "cors.allow-origin": "*"
}
										

$ sudo service elasticsearch restart
										

$ sudo tail -f /var/log/elasticsearch/elasticsearch.log
										

Stop the tail command with Ctrl+C after the restart is complete ("started")

Exercise 17:
AJAX request, second try

$('#search').keyup(function() { // <-- insert after this
  var q = encodeURIComponent($('#search').val());
  $.ajax({
	url: `http://localhost:9200/loc/work/_search?&q=${q}`,
	success: function(result) {
		console.log(JSON.stringify(result));
	},
	error: function(jqXHR, textStatus, errorThrown) {
		console.log(textStatus, ': ', errorThrown);
	}
  });
});
						

(Open your browser's developer tools with Ctrl+Shift+I to see the console.log output)

Response format

{
  "took": 1,
  "timed_out": false,
  "hits": {
    "total": 100,
    "max_score": 1,
    "hits": [
      {
        "_index": "loc",
        "_type": "work",
        "_id": "c000000007",
        "_score": 1,
        "_source": {
          "@context": "context.json",
          "id": "http://id.loc.gov/resources/works/c000000007",
						

Exercise 18: Process hits

success: function(result) {
	console.log(JSON.stringify(result)); // add after this
	$("#result").html('<p/>');
	var hits = result.hits.hits;
	for (let hit of hits) {
		var label = hit._source.label;
		$("#result").append('<hr/>' + label);
	}
}
						

Append the label of each hit to the result div in our HTML

Exercise 19: Improve UI

$("#result").html('<p/>');
										

$("#result").html(`
Hits: ${result.hits.total}
`);
										

$("#result").append('<hr/>' + label);
										

var link = `${label}`;
$("#result").append('
' + link);
										

Show total hits, link from each hit to its JSON

Part IIIb.

Sample searches

Use the page to query the data

Documentation

Which field? How is it nested?

Samples very useful as documentation

Embed a live sample from the index in the search UI

(Read more about how we document the lobid API)

Exercise 20: columns

<div class='row'>
  <div class='col'>
    <code><pre id='sample'></pre></code>
  </div>
  <div class='col'>
    
  </div>
</div>
						

<head>
  <!-- previous head content -->
  <style>
    .row { display: flex; }
    .col { flex: 50%; margin: 5px; }
    #sample { white-space: pre-wrap; }
  </style>
</head>
						

Exercise 21:
Embed sample from index

<script>
 $.ajax({
  url: 'http://localhost:9200/loc/work/c000000026/_source',
  success: function(result) {
    console.log(JSON.stringify(result));
    $("#sample").html(JSON.stringify(result, null, 2));
  },
  error: function(jqXHR, textStatus, errorThrown) {
    console.log(textStatus, ': ', errorThrown);
  }
 });
 // previous script content here
</script>
						

(Open your browser's developer tools with Ctrl+Shift+I to see the console.log output)

★ Comprehensible by introspection

Look at sample document

Full query access to every field you see

(Look up details for meaning of keys in the context)

What you see is what you can query

Don't pre-define fixed params for authors, subjects, etc.

Instead define a general syntax to query any field

New fields added to the data can immediately be queried

Exercise 22: Query syntax

??? # <field>:<term>
										

??? # AND, OR
										

??? # truncate with *
										

??? # <field>:[* TO <???>], creation date?
										

??? # _exists_:<field>, geo information?
										

Elasticsearch query string documentation

Exercise 22: Solutions

label:letters
										

canada OR alaska
										

ameri*
										

adminMetadata.creationDate:[* TO 1995-12-31]
										

_exists_:geographicCoverage
										

Elasticsearch query string documentation

Part IIIc.

Hypothes.is

Embed Hypothes.is in HTML, add annotations to document the data

Hypothes.is

"Annotate the web, with anyone, anywhere"

"Use Hypothesis right now to hold discussions, read socially, organize your research, and take personal notes"

or: to create sample-based, collaborative documentation

Based on browser plugin, proxy server, or embedded JS

Exercise 23:
Embedding Hypothes.is

<head>
  <!-- previous head content -->

  <script type="application/json" class="js-hypothesis-config">
    { "openSidebar": true }
  </script>

  <script src="https://hypothes.is/embed.js" async></script>

  <link rel="canonical" 
    href="https://hbz.github.io/swib18-workshop/search.html"/>
</head>
						

(Only embed.js script is required, config and canonical link are optional)

Exercise 24:
Add annotations

Based on your experience in the query exercise, pick a field and document it like in one of the sample annotations

Markdown syntax is supported for links etc.

Use the `swib18-workshop` tag to make it searchable

https://hypothes.is/search?q=tag:%22swib18-workshop%22

Part III: Use the data to build a web application

Part IV:
Use the data with existing tools

Part IVa.

Kibana

Create visualizations for local index data; look at full data visualizations

Kibana

"Explore, Visualize, Discover Data"

Open source data visualization plugin for Elasticsearch

Running locally at http://localhost:5601/

Exercise 25: Bar chart

Create a bar chart visualization for the 'contribution.agent.type' field

(Kibana will display an additional 'keyword' suffix)

Exercise 26: Pie chart

Create a pie chart visualization for the 'adminMetadata.source.label' field

(Kibana will display an additional 'keyword' suffix)

Exercise 27: Tag cloud

Create a tag cloud visualization for the 'subject.componentList.authoritativeLabel' field

(Kibana will display an additional 'keyword' suffix)

Kibana: so much more

Other types of aggregations

Multiple aggregations in a visualization

Other visualization types

https://www.elastic.co/products/kibana

Kibana:
full LOC index

http://kibana.labs.lobid.org/

Part IVb.

OpenRefine

Use index data in OpenRefine; brief intro to the reconciliation API

OpenRefine

"A powerful tool for working with messy data"

"cleaning it; transforming it from one format into another; and extending it with web services and external data"

Spreadsheet-like user interface, in the browser, locally

Run with ./refine in your OpenRefine location, then open http://localhost:3333/

Exercise 28:
Getting labels for IDs

http://id.loc.gov/resources/works/c000000011
http://id.loc.gov/resources/works/c000000020
http://id.loc.gov/resources/works/c000000026
http://id.loc.gov/resources/works/c000000028
http://id.loc.gov/resources/works/c000000100
						

Copy the lines above to the clipboard

Create URL to get JSON

Exercise 29:
Add first contributing agent

contribution[0].agent.label

Exercise 30:
Getting IDs for labels

Letters from an American farmer
Italian journeys
Report of the trial of George Ryan
The devil upon crutches in England
Educational interests of Colorado
						

Reconciliation API

All of the above: nothing specific built for OpenRefine, just JSON over HTTP, and lots to explore in OpenRefine

Implement OpenRefine Reconciliation API (serve some specific JSON) for better integration in OpenRefine UI

Select best matches using scores and small inline previews, filter by type, suggest fields for data extension, etc.

Reconciliation details

http://blog.lobid.org/2018/08/27/openrefine.html

https://github.com/OpenRefine/OpenRefine/wiki/Reconciliation

http://openrefine.org/documentation.html

Review

Time for questions, discussions, feedback

Question from #eubfws2018

LOUD is more than a nice JSON-LD API

(But a nice JSON-LD API is LOUDer than an RDF file)

Let's recall

Source: Rob Sanderson, "Shout it Out: LOUD", CC-BY

Bibframe, from LOD to LOUD?

Source: Rob Sanderson, "Shout it Out: LOUD", CC-BY

1. Require use cases, with data

Code for exercises and HTML for slides available on the web

https://github.com/hbz/swib18-workshop/