From LOD to LOUD:
building & using JSON-LD APIs

Bootcamp at ELAG 2019

Fabian Steeg / @fsteeg & Adrian Pohl / @acka47 & Pascal Christoph / @dr0ide
Linked Open Data, Hochschulbibliothekszentrum NRW (hbz)

Berlin, 2019-05-07

This presentation:
http://hbz.github.io/elag2019-bootcamp

Overview

Our background & experience

Hochschulbibliothekszentrum des Landes Nordrhein-Westfalen, est. 1973

= Academic library center of North Rhine-Westphalia

Software services for libraries in NRW and beyond

E.g. union catalog, discovery portal DigiBib, ILL, digitization & digital preservation, consortial acquisition

hbz flyer in English (PDF)

lobid: hbz's LOD service, providing bibliographic data, authorities & organizations

Started 2010 with open data publication, triple store & Perl transformation script

Original team: Adrian, Pascal, Felix. Fabian joined in 2012

2013: First API release based on JSON-LD & Elasticsearch

2017–2018: LOUD with lobid API 2.0

Your background & experience

LOUD

Linked Open Usable Data (LOUD)

Source: Rob Sanderson, "Shout it Out: LOUD", CC-BY (video)

Source: Rob Sanderson, "Shout it Out: LOUD", CC-BY (video)

Design for JSON-LD
from RDF, consistently

Part I:
Convert RDF data into usable JSON-LD

Part Ia.

Usable data & APIs

APIs: why, how

Usable data [& APIs]

Libraries are Software

Our collections and services are delivered primarily via software. [...] The choices we make in the development, selection, and implementation of this software [...] define the limits of our content and services. We can only be as good as our software.

— Cody Hanson, Libraries are Software

[Usable data &] APIs

e.g. lobid sources, formats, applications

API: why?

API: why?

API: how?

API: how?

GET
https://lobid.org/organisations/DE-1a

{
  "linkedTo" : {
    "id" : "http://lobid.org/organisations/DE-601#!",
    "label" : "Verbundzentrale des GBV (VZG)"
  },
  "rs" : "110000000000",
  "address" : {
    "addressLocality" : "Berlin",
    "type" : "PostalAddress",
    "addressCountry" : "DE",
    "postalCode" : "10772"
  }
  ...
					

Part Ib.

JSON APIs

Using the lobid-organisations API, processing JSON

Setup

cURL

Command line tool for transferring data with URLs

https://curl.haxx.se/download.html

$ curl --help

Exercise 1: cURL

$ curl "https://lobid.org/organisations/DE-1a"
											

# Copy above, paste in terminal
$ curl "https://lobid.org/organisations/DE-1a"
											

$ ↑ # access the history
$ curl "https://lobid.org/organisations/DE-1a"
											

(To copy & paste from/to GUI applications like your browser use (left!) CTRL+c resp. CTRL+v. You can copy and paste into the terminal with SHIFT+CTRL+c and SHIFT+CTRL+v. )

Response

$ curl https://lobid.org/organisations/DE-1a
{
  "linkedTo" : {
    "id" : "http://lobid.org/organisations/DE-601#!",
    "label" : "Verbundzentrale des GBV (VZG)"
  },
  "rs" : "110000000000",
  "address" : {
    "addressLocality" : "Berlin",
    "type" : "PostalAddress",
    "addressCountry" : "DE",
    "postalCode" : "10772"
  },
...
						

Output is quite long, we often want specific values

JavaScript Object Notation (JSON)

var options = {
  url: 'https://lobid.org/organisations/DE-1a'
};

request(options, function (error, response, body) {
  var doc = JSON.parse(body);
  console.log('postal code:', doc.address.postalCode) // <--
});
						

> postal code: 10772
						

jq

A lightweight and flexible command-line JSON processor

https://stedolan.github.io/jq/

https://jqplay.org/

$ jq --help

Exercise 2a: JSON with jq

$ curl "https://lobid.org/organisations/DE-1a" \
| jq .name # filter: .name
						

Exercise 2b: JSON with jq

$ curl "https://lobid.org/organisations/DE-1a" | jq <???> # get the English label for the type of the lib's funder
										

$ curl "https://lobid.org/organisations/DE-1a" | jq <???> # get the entry's modification date
										

Exercise 2c: JSON with jq

$ curl "https://lobid.org/organisations/DE-1a" | jq .fundertype.label.en
										

$ curl "https://lobid.org/organisations/DE-1a" | jq .mainEntityOfPage.dateModified
										

So JSON is great.

But what if we have RDF data?

Part Ic.

From RDF to JSON

JSON-LD processing with jsonld.js via jsonld-cli

Bibframe works dataset

Example work

JSON-LD

5. Design for JSON-LD, consistently

Great, so let's make JSON-LD from RDF

jsonld-cli

JSON-LD command line interface tool

https://github.com/hbz/jsonld-cli

Our fork of https://github.com/digitalbazaar/jsonld-cli, added import from N-Quads

$ jsonld --help

Exercise 3:
N-Triples to JSON-LD

$ cd ~/git/elag2019-bootcamp/data
										

$ jsonld import loc.nt
										

$ jsonld import loc.nt > loc.json
										

Result

{
 "@id": "http://id.loc.gov/resources/works/c000101650",
 "@type": [
   "http://id.loc.gov/ontologies/bibframe/Work",
   "http://id.loc.gov/ontologies/bibframe/Text"
 ],
 "http://id.loc.gov/ontologies/bibframe/subject": [{
  "@id": "http://id.loc.gov/resources/works/101650#Topic650-20"
 }, ...]
}, {
 "@id": "http://id.loc.gov/resources/works/101650#Topic650-20",
 "http://www.w3.org/2000/01/rdf-schema#label": [{
  "@value": "Climatic changes--Europe."
 }],...
}
					

Serialized RDF as JSON-LD | Specification

Exercise 3a:
Use serialized RDF to access subjects

$ cat loc.json
										

# like .address.postalCode, but for subject labels?
$ cat loc.json | jq <???>
										

So JSON is great?

Result

{
 "@id": "http://id.loc.gov/resources/works/c000101650",
 "@type": [
   "http://id.loc.gov/ontologies/bibframe/Work",
   "http://id.loc.gov/ontologies/bibframe/Text"
 ],
 "http://id.loc.gov/ontologies/bibframe/subject": [{
  "@id": "http://id.loc.gov/resources/works/101650#Topic650-20"
 }, ...]
}, {
 "@id": "http://id.loc.gov/resources/works/101650#Topic650-20",
 "http://www.w3.org/2000/01/rdf-schema#label": [{
  "@value": "Climatic changes--Europe."
 }],...
}
					

Serialized RDF as JSON-LD | Specification

Data processing vs. API access

for o in doc :
		if o["@type"] contains "http://id.loc.gov/ontologies/bibframe/Work" :
			subjectId = o["http://id.loc.gov/ontologies/bibframe/subject"][0]["@id"] // + error handling

for o in doc :
	if o["@id"] == subjectId :
		subjectLabel = o["http://www.w3.org/2000/01/rdf-schema#label"]["@value"] // + error handling
										

doc.address.postalCode // doc.subject.label?
										

Data structures

If you've chosen the right data structures and organized things well, the algorithms will almost always be self­evident. Data structures, not algorithms, are central to programming.

Developer empathy

Non-developer empathy

★ Few barriers to entry

Design for JSON-LD from RDF, consistently

Framing

Exercise 4:
Framing

{
 "@type": "http://id.loc.gov/ontologies/bibframe/Work",
 "@embed": "@always"
}
										

$ jsonld frame -f frame.json loc.json
										

$ jsonld frame -f frame.json loc.json > loc-framed.json
										

Result

{
 "@id": "http://id.loc.gov/resources/works/c000101650",
 "http://id.loc.gov/ontologies/bibframe/subject": [{
   "@id":"http://id.loc.gov/resources/works/101650#Topic650-20",
   "@type": [
     "http://id.loc.gov/ontologies/bibframe/Topic",
     "http://www.loc.gov/mads/rdf/v1#ComplexSubject"
   ],
   ...
   "http://www.w3.org/2000/01/rdf-schema#label": "Climatic changes--Europe."
 }, {...}],
}
					

Framed JSON-LD | Specification

Exercise 4a:
Use framed JSON-LD to access subjects

$ cat loc-framed.json
										

# like .address.postalCode, but for subject labels?
$ cat loc-framed.json | jq <???>
										

2. As simple as possible?

JSON-LD Context

How do we get the context?

Context: more than simple mappings

Consistency

Array as default

"contribution": {
  "@id": "http://id.loc.gov/ontologies/bibframe/contribution",
  "@container": "@set"
}
						

Simplicity: move definitions from data to context

"creationDate": {
  "@id": "http://id.loc.gov/ontologies/bibframe/creationDate",
  "@type": "http://www.w3.org/2001/XMLSchema#date"
}
						

Exercise 5:
Compact

$ jsonld compact -c context.json loc-framed.json
										

$ jsonld compact -c context.json loc-framed.json > loc-compact.json
										

Result

{
 "@id": "http://id.loc.gov/resources/works/c000101650",
 "subject": [
  {
   "id": "http://id.loc.gov/resources/works/101650#Topic650-20",
   "type": [
    "Topic",
    "ComplexSubject"
   ],
   "label": "Climatic changes--Europe."
  }
 ], ...
}
					

Compacted JSON-LD | Specification

Exercise 5a:
Use compact JSON-LD to access subjects

$ cat loc-compact.json
										

$ cat loc-compact.json | jq .label
										

# like .address.postalCode, but for subject labels?
$ cat loc-compact.json | jq <???>
										

Exercise 5a:
Solution

$ cat loc-compact.json | jq .subject.label
jq: error: Cannot index array with string "label"
										

$ cat loc-compact.json | jq .subject[0].label
										

$ cat loc-compact.json | jq .subject[].label
										

Missing values in context

Exercise 5b: Fix context

$ cat loc-compact.json | jq -c 'path(..)'
										

$ cat loc-compact.json | jq -c 'path(..)' | grep http
										

"value" : "http://www.w3.org/1999/02/22-rdf-syntax-ns#value"
										

$ jsonld compact -c context.json loc-framed.json > loc-compact.json
										

Exercise 6: Use final JSON

$ cat loc-compact.json
										

$ cat loc-compact.json | jq <???> # get authors
										

$ cat loc-compact.json | jq <???> # get subtitle
										

$ jsonld format -q data/loc-compact.json
										

Exercise 6: Solutions

$ cat loc-compact.json | jq .contribution[].agent.label
										

$ cat loc-compact.json | jq .title.subtitle
										

So JSON is great

RDF view

1. Require use cases, with data

JSON view

Context maps stable API to implementation details

JSON with context

(based on part of https://api.github.com/users/hbz)

{
  "@id": "https://github.com/users/hbz",
  "type": "Organization",
  "name": "hbz",
  "blog": "https://www.hbz-nrw.de",

  "@context": {
    "type": "@type",
    "name": "https://schema.org/name",
    "blog": { "@id": "https://schema.org/url", "@type": "@id" },
    "Organization": "https://schema.org/Organization"
  }
}
					

You can test this in the JSON-LD Playground

JSON as RDF

{
  "@context": "...",
  "type": "Organization",
  "name": "hbz",
  "blog": "https://www.hbz-nrw.de",
}
					

<https://github.com/users/hbz> \
<http://www.w3.org/1999/02/22-rdf-syntax-ns#type> \
<https://schema.org/Organization> .
<https://github.com/users/hbz> <https://schema.org/name> "hbz" .
<https://github.com/users/hbz> <https://schema.org/url> \
<https://www.hbz-nrw.de> .
					

Output of N-Quads tab in the JSON-LD Playground

Part I: Convert RDF data into usable JSON-LD

Part II:
Index and access the data with Elasticsearch

Part IIa.

Elasticsearch bulk indexing

Create bulk format with jq, index with curl

Elasticsearch

Exercise 7: Convert more data (100 documents)

$ tail loc-100.nt
										

$ jsonld import <???>
										

$ jsonld frame <???>
										

$ jsonld compact <???>
										

Exercise 7: Solutions

$ jsonld import loc-100.nt > loc-100.json
										

$ jsonld frame -f frame.json loc-100.json > loc-100-framed.json
										

$ jsonld compact -c context.json loc-100-framed.json > loc-100-compact.json
										

Elasticsearch bulk format

{"index":{"_index":"loc","_type":"work","_id":"c000101650"}}

{@context: ...,"id":"http://...","type":["Work","Text"],...}

Input data format

{
  "@context": { ... },
  "@graph": [
    { "id": "http://.../1", "type": ["Work", "Text"], ... },
    { "id": "http://.../2", "type": ["Work", "Text"], ... },
    ...
  ]
}
							

Exercise 8a: Document line

$ cat loc-100-compact.json | jq
										

$ cat loc-100-compact.json | jq -c .
										

$ cat loc-100-compact.json | jq -c '.["@graph"][]'
										

@graph requires special escaping in jq with ', [], "

Exercise 8b: Metadata line

$ cat loc-100-compact.json | \
jq -c '.["@graph"][] | .'
									

$ cat loc-100-compact.json | \
jq -c '.["@graph"][] | { test:1 }'
										

$ cat loc-100-compact.json | \
jq -c '.["@graph"][] | { _id:.id }'
										

$ cat loc-100-compact.json | \
jq -c '.["@graph"][] | { "index":{"_index":"loc", "_type":"work", "_id":.id} }'
										

Exercise 8c: Putting it together

$ cat loc-100-compact.json | jq -r \
'.["@graph"][] | "\({meta:1})\n\({data:2})"'
										

$ cat loc-100-compact.json | jq -r \
'.["@graph"][] | "\({ "index":{"_index":"loc", "_type":"work", "_id":.id} })\n\(.)"' \
> bulk.ndjson
										

Elasticsearch bulk API

Setup

You Know,
for Search

$ curl localhost:9200

Exercise 9: Bulk indexing

$ tail -n 2 bulk.ndjson
										

$ curl -H "Content-Type: application/x-ndjson" \
-XPOST localhost:9200/_bulk \
--data-binary "@bulk.ndjson" | jq
										

$ curl localhost:9200/loc/work/_search | jq
										

Part IIb.

Access the Elasticsearch index

Search queries, direct access by ID

Elasticsearch URI search

Exercise 10: Query syntax

$ curl localhost:9200/loc/work/_search?q=<???> | jq
										

# <field>:<term>
$ curl localhost:9200/loc/work/_search?q=<???> | jq
										

# AND, OR
$ curl localhost:9200/loc/work/_search?q=<???> | jq
										

# truncate with *
$ curl localhost:9200/loc/work/_search?q=<???> | jq
										

Exercise 10: Solutions

$ curl localhost:9200/loc/work/_search?q=america | jq
										

$ curl localhost:9200/loc/work/_search?q=subject.label:education | jq
										

$ curl localhost:9200/loc/work/_search?q=canada+OR+alaska | jq
										

$ curl localhost:9200/loc/work/_search?q=ameri* | jq
										

Direct access by ID

Exercise 11a: Re-index

$ cat loc-compact.json | jq '(.id / "/") | last'
										

$ cat loc-100-compact.json | jq -r \
'.["@graph"][] | "\({ "index":{"_index":"loc", "_type":"work", "_id":(.id / "/") | last} })\n\(.)"' > bulk.ndjson
										

$ curl -XDELETE localhost:9200/loc | jq
										

$ curl -H "Content-Type: application/x-ndjson" \
-XPOST localhost:9200/_bulk \
--data-binary "@bulk.ndjson" | jq
										

Exercise 11b: GET by ID

$ curl "localhost:9200/<???>/<???>/<???>" \
| jq
										

$ curl \
"localhost:9200/<???>/<???>/<???>/_source" \
| jq
										

Exercise 11c: Solutions

$ curl "localhost:9200/loc/work/c000000026" \
| jq
										

$ curl "localhost:9200/loc/work/c000000026/_source" \
| jq
										

Direct access by ID

Exercise 12a: Remote JSON to N-Quads

$ jsonld format -q loc-compact.json
										

$ jsonld format -q http://localhost:9200/loc/work/c000000026/_source
										

Input data format

{
  "@context": { ... },
  "@graph": [
    { "id": "http://.../1", "type": ["Work", "Text"], ... },
    { "id": "http://.../2", "type": ["Work", "Text"], ... },
    ...
  ]
}
						

We processed & indexed each element of the @graph array

Add context

Exercise 12b: Serve context

$ cd ~/git/elag2019-bootcamp ; npm install ; npm start
										

$ curl http://localhost:3000/context.json | jq
										

Exercise 12c: Link to context

$ cat loc-100-compact.json | \
jq -c '.["@graph"][] | . + {"test": 1}'
										

$ cat loc-100-compact.json | \
jq -c '.["@graph"][] | . + {"@context":"http://localhost:3000/context.json"}'
										

$ cat loc-100-compact.json | jq -r \
'.["@graph"][] | "\({ "index":{"_index":"loc", "_type":"work", "_id":(.id / "/") | last} })\n\({"@context":"http://localhost:3000/context.json"}+.)"' > bulk.ndjson
										

Exercise 12d: Reindex and use

$ curl -XDELETE localhost:9200/loc | jq
										

$ curl -H "Content-Type: application/x-ndjson" \
-XPOST localhost:9200/_bulk \
--data-binary "@bulk.ndjson" | jq
										

$ jsonld format -q http://localhost:9200/loc/work/c000000026/_source
										

API: How — JSON over HTTP

Serving our data as JSON over HTTP, serializable as RDF

Elasticsearch: so much more

Boosting, Analyzers, Aggregations, etc.

Elasticsearch: The Definitive Guide (concepts)

Elasticsearch Reference (current API)

Part II: Index and access the data with Elasticsearch

Part III:
Use the data to build a web application

Part IIIa.

Search page

HTML page for index search

Exercise 13: HTML

$ cd data; gedit search.html &
										

<!DOCTYPE html>
<html>
	<head>
		<meta charset="UTF-8">
		<title>Title</title>
	</head>
	<body>
		Hello HTML
	</body>
</html>
										

$ chromium-browser search.html &
										

JQuery

Exercise 14: Text Input

<head>
  <!-- previous head content -->
  <script src="https://code.jquery.com/jquery-3.3.1.min.js">
  </script>
</head>
						

<body>
	<input id='search' type='text'/>
	<script>
		$('#search').keyup(function() {
			var q = $('#search').val();
			console.log(q);
		});
	</script>
	<div id='result'></div>
</body>
						

(Open your browser's developer tools with Ctrl+Shift+I to see the console.log output)

AJAX

Exercise 15: AJAX request

$('#search').keyup(function() { // <-- insert after this
  var q = encodeURIComponent($('#search').val());
  $.ajax({
	url: `http://localhost:9200/loc/work/_search?&q=${q}`,
	success: function(result) {
		console.log(JSON.stringify(result));
	},
	error: function(jqXHR, textStatus, errorThrown) {
		console.log(textStatus, ': ', errorThrown);
	}
  });
});
						

(Open your browser's developer tools with Ctrl+Shift+I to see the console.log output)

Cross-Origin Resource Sharing (CORS)

Exercise 16: Configure CORS

$ sudo gedit /etc/elasticsearch/elasticsearch.yml
										

"http": {
    "cors.enabled": true,
    "cors.allow-origin": "*"
}
										

$ sudo service elasticsearch restart
										

$ sudo tail -f /var/log/elasticsearch/elasticsearch.log
										

Stop the tail command with Ctrl+C after the restart is complete ("started")

Exercise 17:
AJAX request, second try

$('#search').keyup(function() { // <-- insert after this
  var q = encodeURIComponent($('#search').val());
  $.ajax({
	url: `http://localhost:9200/loc/work/_search?&q=${q}`,
	success: function(result) {
		console.log(JSON.stringify(result));
	},
	error: function(jqXHR, textStatus, errorThrown) {
		console.log(textStatus, ': ', errorThrown);
	}
  });
});
						

(Open your browser's developer tools with Ctrl+Shift+I to see the console.log output)

Response format

{
  "took": 1,
  "timed_out": false,
  "hits": {
    "total": 100,
    "max_score": 1,
    "hits": [
      {
        "_index": "loc",
        "_type": "work",
        "_id": "c000000007",
        "_score": 1,
        "_source": {
          "@context": "context.json",
          "id": "http://id.loc.gov/resources/works/c000000007",
						

Exercise 18: Process hits

success: function(result) {
	console.log(JSON.stringify(result)); // add after this
	$("#result").html('<p/>');
	var hits = result.hits.hits;
	for (let hit of hits) {
		var label = hit._source.label;
		$("#result").append('<hr/>' + label);
	}
}
						

Append the label of each hit to the result div in our HTML

Exercise 19: Improve UI

$("#result").html('<p/>');
										

$("#result").html(`
Hits: ${result.hits.total}
`);
										

$("#result").append('<hr/>' + label);
										

var link = `${label}`;
$("#result").append('
' + link);
										

Show total hits, link from each hit to its JSON

Part IIIb.

Sample searches

Use the page to query the data

Documentation

Which field? How is it nested?

Samples very useful as documentation

Embed a live sample from the index in the search UI

(Read more about how we document the lobid API)

Exercise 20: columns

<div class='row'>
  <div class='col'>
    <code><pre id='sample'></pre></code>
  </div>
  <div class='col'>
    
  </div>
</div>
						

<head>
  <!-- previous head content -->
  <style>
    .row { display: flex; }
    .col { flex: 50%; margin: 5px; }
    #sample { white-space: pre-wrap; }
  </style>
</head>
						

Exercise 21:
Embed sample from index

<script>
 $.ajax({
  url: 'http://localhost:9200/loc/work/c000000026/_source',
  success: function(result) {
    console.log(JSON.stringify(result));
    $("#sample").html(JSON.stringify(result, null, 2));
  },
  error: function(jqXHR, textStatus, errorThrown) {
    console.log(textStatus, ': ', errorThrown);
  }
 });
 // previous script content here
</script>
						

(Open your browser's developer tools with Ctrl+Shift+I to see the console.log output)

★ Comprehensible by introspection

Look at sample document

Full query access to every field you see

(Look up details for meaning of keys in the context)

What you see is what you can query

Don't pre-define fixed params for authors, subjects, etc.

Instead define a general syntax to query any field

New fields added to the data can immediately be queried

Exercise 22: Query syntax

??? # <field>:<term>
										

??? # AND, OR
										

??? # truncate with *
										

??? # <field>:[* TO <???>], creation date?
										

??? # _exists_:<field>, geo information?
										

Elasticsearch query string documentation

Exercise 22: Solutions

label:letters
										

canada OR alaska
										

ameri*
										

adminMetadata.creationDate:[* TO 1995-12-31]
										

_exists_:geographicCoverage
										

Elasticsearch query string documentation

Part IIIc.

Hypothes.is

Embed Hypothes.is in HTML, add annotations to document the data

Hypothes.is

"Annotate the web, with anyone, anywhere"

"Use Hypothesis right now to hold discussions, read socially, organize your research, and take personal notes"

or: to create sample-based, collaborative documentation

Based on browser plugin, proxy server, or embedded JS

Exercise 23:
Embedding Hypothes.is

<head>
  <!-- previous head content -->

  <script type="application/json" class="js-hypothesis-config">
    { "openSidebar": true }
  </script>

  <script src="https://hypothes.is/embed.js" async></script>

  <link rel="canonical"
    href="https://hbz.github.io/elag2019-bootcamp/search.html"/>
</head>
						

(Only embed.js script is required, config and canonical link are optional)

Exercise 24:
Add annotations

Based on your experience in the query exercise, pick a field and document it like in one of the sample annotations

Markdown syntax is supported for links etc.

Use the `elag2019-bootcamp` tag to make it searchable

https://hypothes.is/search?q=tag:%22elag2019-bootcamp%22

Part III: Use the data to build a web application

Part IV:
Use the data with existing tools

Part IVa.

Kibana

Create visualizations for local index data; look at full data visualizations

Kibana

"Explore, Visualize, Discover Data"

Open source data visualization plugin for Elasticsearch

Running locally at http://localhost:5601/

Exercise 25: Bar chart

Create a bar chart visualization for the 'contribution.agent.type' field

(Kibana will display an additional 'keyword' suffix)

Exercise 26: Pie chart

Create a pie chart visualization for the 'adminMetadata.source.label' field

(Kibana will display an additional 'keyword' suffix)

Exercise 27: Tag cloud

Create a tag cloud visualization for the 'subject.componentList.authoritativeLabel' field

(Kibana will display an additional 'keyword' suffix)

Kibana: so much more

Other types of aggregations

Multiple aggregations in a visualization

Other visualization types

https://www.elastic.co/products/kibana

Kibana:
full LOC index

http://kibana.labs.lobid.org/

Part IVb.

OpenRefine

Use index data in OpenRefine; brief intro to the reconciliation API

OpenRefine

"A powerful tool for working with messy data"

"cleaning it; transforming it from one format into another; and extending it with web services and external data"

Spreadsheet-like user interface, in the browser, locally

Run with ./refine in your OpenRefine location, then open http://localhost:3333/

Exercise 28:
Getting labels for IDs

http://id.loc.gov/resources/works/c000000011
http://id.loc.gov/resources/works/c000000020
http://id.loc.gov/resources/works/c000000026
http://id.loc.gov/resources/works/c000000028
http://id.loc.gov/resources/works/c000000100
						

Copy the lines above to the clipboard

Create URL to get JSON

Exercise 29:
Add first contributing agent

contribution[0].agent.label

Exercise 30:
Getting IDs for labels

Letters from an American farmer
Italian journeys
Report of the trial of George Ryan
The devil upon crutches in England
Educational interests of Colorado
						

Reconciliation API

All of the above: nothing specific built for OpenRefine, just JSON over HTTP, and lots to explore in OpenRefine

Implement OpenRefine Reconciliation API (serve some specific JSON) for better integration in OpenRefine UI

Select best matches using scores and small inline previews, filter by type, suggest fields for data extension, etc.

Reconciliation details

blog.lobid.org/2018/08/27/openrefine.html

github.com/OpenRefine/OpenRefine/wiki/Reconciliation

openrefine.org/documentation.html

Review

Time for questions, discussions, feedback

Code for exercises and HTML for slides:

https://github.com/hbz/elag2019-bootcamp/

Blog post "Indexing the Bibframe works dataset as JSON-LD":

http://blog.lobid.org/2019/01/31/bibframe-dataset.html