text to come here
Thursday, 9 January 2025
Sunday, 17 September 2023
Setting up the revised Collation Editor: file structures
In an earlier post, I explained some of the history behind the Collation Editor, and our use of it in Textual Communities. At last, I am updating the Collation Editor embedded into TC!
The Collation Editor has two major dependencies:
- On Python, for a series of critical tasks run through a Python server;
- On CollateX, for the actual collation.
The first task was to create a version of the Collation Editor Core implementing both dependencies. I did this by mirroring the structure of the stand-alone collation editor code (available at https://github.com/itsee-birmingham/standalone_collation_editor). Thus, this is what the top-level folder looks like in my implementation (in my installation, in /Applications/Collation_Editor_Core):
That is: at the root level I have a folder holding collateX, with the collatex-tools jar in it. There is a folder labelled "collation" which we will look at in a moment. There are two python files, and then a .sh and .bat file which start up the application (this structure is taken from the current stand-alone collation editor structure).
Within the "collation" folder, here is what I have:
You see here a series of .py files, all needed for the link to Python to work. However, we need to have an index.html file in place to run the instance. The index.html file is actually contained within the "collation/static" folder, as follows:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><meta http-equiv="X-UA-Compatible" content="IE=8" /><title>Collation Editor</title><meta name="description" content="Collation and Apparatus Editor" /><meta http-equiv="X-UA-Compatible" content="IE=edge" /><script>var SITE_DOMAIN = "http://localhost:8080";var staticUrl = SITE_DOMAIN + '/collation/';</script><script type="text/javascript" src="/collation/js/jquery-3.3.1.min.js"></script><script type="text/javascript" src="/collation/js/jquery-ui.min.js"></script><link rel=stylesheet href="/collation/pure-release-1.0.0/pure-min.css"type="text/css"/><script type="text/javascript" src="/collation/CE_core/js/collation_editor.js"></script><script type="text/javascript">var servicesFile = 'js/local_services.js';collation_editor.init();</script></head><body oncontextmenu="return false;"><div id="header" class="collation_header"><h1 id="stage_id">Collation</h1><h1 id="project_name"></h1><div id="login_status"></div></div><div id="container"><p>Loading, Please wait.</p><br/><br/></div><div id="footer"></div><div id="tool_tip" class="tooltip"></div></body></html>
Note that the "src" and "href" attributes direct to "/collation..." not to "collation..". The preceding "/" is important as this sends the server to look for these files in the root "collation" folder.
With this structure we can start up an instance of the Collation Editor with both Python and CollateX in place by going to the terminal, moving into the root directory thus:
cd /Applications/Collation_Editor_Core
And then starting up the instance with
./startup.sh
This calls Python 3 to start a server at localhost:8080, with the "collation" folder as the root, and running the Python .py files in the "collation/core" folder. It also starts up CollateX, from the "collatex" folder at the root, with CollateX running on another. If all is in place, this is what you will see when you go to "http://localhost:8080/collation/" in your browser.
If you have the "data" folder from the stand-alone installation in the "collation" folder, you can type in "B04K6V23" into the "Select" box and then hit the "Collate Project Witnesses" button (currently not working ...)
Setting up the revised Collation editor: some history (2023)
I am a huge fan of the "Collation Editor", built by Cat Smith of the Institute for Textual Scholarship and Electronic Editing (ITSEE) at the University of Birmingham, with substantial input from Troy Griffitts, now at The Göttingen Academy of Sciences and Humanities in Lower Saxony. Some history is required. The roots of the Collation Editor lie in my Collate software, written for the Macintosh computer from 1989 on and, in its day, used heavily by multiple editing projects. Notable among these user projects were two groups editing Biblical texts: those associated with the Institute for New Testament research at Münster, Germany (INTF), and David Parker and scholars working with him at the University of Birmingham (now, ITSEE).
Part of the story of how Collate begat CollateX, and CollateX begat the Collation Editor, is told in other blogs on this site: https://scholarlydigitaleditions.blogspot.com/2014/09/the-history-of-collate.html and https://scholarlydigitaleditions.blogspot.com/2014/09/collate-2-and-design-for-its-successor.html. These blogs, though here dated 2014, were written in 2007. Other parts can be deduced from an article about the evolution of digital methods in the INTF and ITSEE written by myself, David Parker, Hugh Houghton and Klaus Wachtel (you can read that article at my Academia site, or via its DOI).
The first part of this begetting is the making of CollateX. CollateX fulfilled completely the first part of the agenda I laid out in the blogs on this site: to create a system for comparison of multiple texts which was modular and independent of any one hardware or software implementation. CollateX is a marvel, and a remarkable achievement by the team of software engineers who made it (prominently, Ronald Dekker of the Huygens Institute, Amsterdam).
The second part of this begetting was the making of the Collation Editor. This creates an entire environment permitting editors to create exactly the collation they want, by determining through a point-and-click interface exactly what words collate with what and how the collation is to be expressed. Essentially, the Collation Editor is an interface to, and an extension of, CollateX: permitting editors to adjust the CollateX collations to create exactly the collations they want. For me, the test of the Collation Editor, and its implementation of CollateX, was simple: could we achieve exactly the same complex collations with the Collation Editor/CollateX as we could, from 1995 to around 2015, with Collate? The answer is, triumphantly, yes. Indeed, we could achieve far more with the Collation Editor than we ever could with Collate. Here is the tool I dreamed of in 2007. (Somewhere, I said that it would take a team of ten people ten years to make the replacement for Collate. I was not far wrong).
Accordingly, in 2016 I started work on integrating the Collation Editor into Textual Communities. We have now used this integrated implementation to collate some four thousand lines of the Canterbury Tales, in preparation of our forthcoming Critical Edition of the Tales. You can see how this works in a video I made, collating just one line of the Tales. As you can see, the Collation Editor can create exactly the highly-complex collations we want. In the last years, it has become an absolutely vital part of our work on the Tales. However, the version we integrated in 2016, and which is still the version we are using, is now seriously outdated. Many improvements have been made to the Collation Editor since 2016 (or, in effect, 2019, when we last updated our implementation of the Collation Editor) and finally, thanks to a sabbatical, I am setting out to bring the Textual Communities version of the Collation Editor up to date. This task should be greatly eased by the re-organization and rewriting of the Collation Editor since 2019. The Collation Editor code has now been cleanly divided into a "core" code library, designed so that the whole core can fit inside any implementation and be easily updated, and a "services" code library, which connects the core to whatever implementation you want. In our case, we use MongoDB document databases to store all our information about our texts, and hence everything the Collation Editor needs to function should be linked to our MongoDB databases.
In the next posts, I will explain how I went about setting up the updated core collation tools of the Colllation Editor to work within Textual Communities, in the same way as a series of blogs on StaticSearch explain how I got this to work with our data.
Sunday, 6 August 2023
In praise of staticSearch
Over the last few weeks, I have worked intensively with staticSearch to integrate it into our forthcoming publication of Edvige Agostinelli and Bill Coleman's digital edition of Boccaccio's Teseida. You can see the near-complete prototype at http://inklesseditions.com/TeseidaStatic/. Note that this is still a prototype only, and the address will change as we move to full publication; please DO NOT repost this link on the open internet. Among other matters, this is "Endings"-conformant in its principal components: see https://scholarlydigitaleditions.blogspot.com/2023/07/the-endings-project-and-canterbury.html.
As you can see (try searching for "come" by typing it in the search box and hitting return, or the search icon) staticSearch works beautifully here. And hence my final word on staticSearch. This is a quite wonderful tool. It is lightning fast, easy to set up, and works like a dream. As a true Endings tool, it has no dependencies on any outside systems of any kind. Take a bow, Martin Holmes and Joey Takeda (and everyone else who has contributed). Great work.
Sunday, 30 July 2023
Setting up staticSearch for our projects: nested files and multiple search entry points
I now realize (a week later!), after looking at the staticSearch projects listed in the documentation two things I did not know before, two things where our projects differ (it seems) from all other StaticSearch implementations to date:
- staticSearch assumes (or at least, all the listed projects appear to follow this model) that all the pages to be searched are held in the same folder as the root index.html folder. Indeed, a 2019 presentation by the staticSearch team explictly declares that "All pages live together in the same folder" and, furthermore, "We don't care" if that means there are 10,591 files in that one folder.
- staticSearch assumes (or at least, all the listed projects appear to follow this model) that all searches are launched from a single place, and a single file, contained in that same folder holding all the project files.
In fact, we discovered that the '<recurse>true</recurse>' statement in the configuration files means that staticSearch has no problem at all with nested directories. It duly finds and indexes all our html pages. But the second issue -- that the default staticSearch configuration expects that all searches will be run from a single file, located in the project home directory -- does cause problems. We could, quite easily, have set up our projects the same way as staticSearch expects, so that clicking on a "search" icon or similar on each of the 90,000 pages would send the reader to a single search page, presumably in the home directory. But we did not want to do that. Here is how the header for one of our project pages looks (for folio 72r of the Naples manuscript of the forthcoming Agostinelli/Coleman edition of Boccaccio's Teseida looks:
A fundamental principle of our edition design is "have only the pages you really need". We want our readers to be able to run the search directly from the page they are looking at, and not have to go to any other page to do the search. Further, we want the header on all our pages to look the same, following another mantra: "keep everything as uniform as possible across the whole edition". This meant that every one of our (possibly) 90,000 html pages would have a search box on it, as you can see in the top right of this image. This means too that searches would not always begin from a file located at the root of project folder. Indeed, all searches except those run from the index.html starting point to our editions would begin from a file nested four layers deep in the project folder. And that is why we found the problems with folder paths referred to in the previous post.
I will post a suggestion in the staticSearch issues forum as to how staticSearch itself could help projects configured like ours, with many files spread over multiple folders and each file being a search access point. In a final post in this series, I offer some general thoughts about staticSearch.
Monday, 24 July 2023
Setting up staticSearch for our project: the search results
In the last post, we integrated the search page into the header of our pages. Now, we need to deal with the search results. staticSearch by default places the search results in the <div id="ssResults"> element. But as part of our set up, we hid that element. Instead, we want the search results to appear in a different place on the page: in a <div id="searchContainer">. So how do we do this?
staticSearch has anticipated that users might want to intervene at the end of the search process to adjust how and where the search results appear. You can find this out by digging into the ssSearch-debug.js file which staticSearch helpfully makes available (it is in the staticSearch folder which Ant makes in your project folder). In it you will find references to a "searchFinishedHook" function, created explicitly as a hook where developers such as me can get at the results of the search and manipulate them before they are seen by the user. The definition of searchFinishedHook is left open in the staticSearch initialization:
this.searchFinishedHook = function(num){};
Accordingly we can redefine searchFinishedHook to let us do what we want to the search results. In this piece of code in the ssInitialize.js file in the staticSearth folder, we define our own searchFinishedHook function:
window.addEventListener('load', function() {Sch = new StaticSearch();
Sch.searchFinishedHook = function (num) {
$("#splash").hide();
$("#rTable").hide();
$("#searchContainer").html($("#ssResults").html());
$("#searchContainer").show();
}
});
The first two lines hide the "splash" and "rTable" elements on the page. The next line copies all the search results for the hidden "ssResults" element into the "searchContainer" element, and the last line shows that element. Here is what it looks like for a simple search:
For a first, straight out-of-the-box effort, this is really impressive! And also lightning fast. You can see too that staticSearch has set us up with links to each page. In our project, we have the base index.html at the root of our project folder. All the files with transcripts of each page are in a folder labelled "html", with a subfolder "transcripts', a subfolder for each manuscript, and finally the html for each page. Thus the transcript for folio 1r of manuscript AUT is in html/transcripts/AUT/1r.html.
staticSearch knows that at each page it indexes is in the folder "html/transcripts/..." relative to the index.html file. Accordingly, staticSearch creates a link to each page as (for example) href="html/transcripts/AUT/1r.html". The links to files works fine for searches from our index.html file. Following standard web protocols, the path "html/transcripts/AUT/1r.html" is appended to the path to the index.html file (e.g. "https://www.inklesseditions.com/teseida/," thus becoming "https://www.inklesseditions.com/teseida/html/transcripts/AUT/1r.html".
However, we want to do searches not just from the root index.html file, but also from every transcript file. That is: the calling file for each search is NOT at the root directory, where the index.hmtl file is, but in a directory nested several layers deep within the root. For example, a search from the transcript file for page 1r of the AUT manuscript will be sent from html/transcripts/AUT/1r.html. This affects all the file calls needed and created for staticSearch. Accordingly, we have to make multiple adjustments within the file 1r.html to prepare it for static search:
The calls to ssinitialize.js and ssSearch.js have to go to ../../../staticSearch/ssinitialize.js and
../../../staticSearch/ssSearch.js (the same for ssSearch-debug.js if you are using that instead)
The attribute value @data-ssfolder on the form id="ssForm" which runs the search has to be set
to ../../../staticSearch and not just staticSearch
With these adjustments, the search from 1r.html runs just fine. But we have a problem with the links to the pages in the search results. It is this: because staticSearch does not understand that the 1r.html file is buried in html/transcripts/AUT/ it fails to create valid links in the search results to the files containing the search hits. For example: the link from html/transcripts/AUT/1r.html to 1r.html of the NO manuscript should be html/transcripts/NO/1r.html. Instead, StaticSearch links to
html/transcripts/AUT/html/transcripts/NO/1r.html
That is: it concatenates the file path for html/transcripts/NO/1r.html with the path for html/transcripts/AUT/. The path should actually be '../../../html/transcripts/NO/1r.html'.
It is not too difficult to fix these incorrect paths by calling a function to rewrite these internal links in our override of the searchFinishedHook function. But this is somewhat ugly. A better solution would be to have staticSearch recognize where it is functioning from in the file-system and adjust links accordingly in the Ant process. In the next post, I explore how these problems have arisen and what might be done about it.
Sunday, 23 July 2023
Setting up staticSearch for our project: integrating the search box into our pages
In earlier posts, I describe the background to the decision to use staticSearch and my experience of getting it to work. In this post, I describe how we are winding staticSearch into our editions.
By default, staticSearch places everything it uses into a <div id="staticSearch"> element. So your core search page, typically the "index.html" file at the root of your document collection, has to contain a <div id="staticSearch"> </div> element. When you run the Ant process, as described in https://scholarlydigitaleditions.blogspot.com/2023/07/staticsearch-and-me.html, the <div id="staticSearch"> gets populated with multiple javascript and html statements. Here is the beginning of what staticSearch pastes in, as of version 1.4.4:
<div id="staticSearch">
<script xmlns="http://www.w3.org/1999/xhtml" src="staticSearch/ssSearch-debug.js"></script>
<script xmlns="http://www.w3.org/1999/xhtml" src="staticSearch/ssInitialize.js"></script>
<noscript xmlns="http://www.w3.org/1999/xhtml">This page requires JavaScript.</noscript>
<form xmlns="http://www.w3.org/1999/xhtml" accept-charset="UTF-8" id="ssForm"
data-allowphrasal="yes" data-allowwildcards="yes" data-minwordlength="2"
data-scrolltotextfragment="no" data-maxkwicstoshow="5" data-resultsperpage="5"
onsubmit="return false;" data-versionstring="" data-ssfolder="../../../staticSearch"
data-kwictruncatestring="..." data-resultslimit="2000">
<span class="ssQueryAndButton">
<input type="text" id="ssQuery" aria-label="Search"/>
<button id="ssDoSearch">Search</button>
</span>
</form>
This fragment sets up the search form, which will appear where-ever you have put <div id="staticSearch"> in your document. In our implementation, we place it in the document header, where we want this to take up very little space. In the default implementation, the <form id="ssForm"> is followed by two other elements, thus:
<div xmlns="http://www.w3.org/1999/xhtml" id="ssSearching" >Searching...</div>
<div xmlns="http://www.w3.org/1999/xhtml" id="ssResults"></div>
<div xmlns="http://www.w3.org/1999/xhtml" id="ssPoweredBy"> //ssLogo etc
If we keep this as is, here is what the top of our page looks like:
This is rather ugly, as "ssSearching" and ""ssPoweredBy" elements intrude on our very clear header. So we can suppress those by adding 'style="display:none"' to those elements. We will also add 'style="display:none"' to the "ssResults" element: more on that in a moment. Thus:
<div xmlns="http://www.w3.org/1999/xhtml" id="ssSearching"
style="display:none" >Searching...</div>
Now, this is how it looks once those elements have been hidden:
This could be better yet. Space in the header is at a premium: every letter counts. Instead of "Search", taking up rather a large box, we might have just a "?" or, even better, a search icon.S So we adjust the appearance of the "Search" button with this bit of css:
#ssDoSearch {
background-image: url("../../common/images/searchicon.png");
background-size: 15px;
height: 24px;
width: 23px;
background-position: 50%;
background-repeat: no-repeat;
position: relative;
top: 5px;
}
#ssQuery {
width: 100px;
}
The #ssQuery also makes the search box a bit narrower. So now it looks so:
This is beginning to look fine. Now, let's see what the search results look like in the next post.
Thursday, 20 July 2023
staticSearch and Me: getting started
In another post, I explain the background to my work in making digital scholarly editions in relation to the Endings project, and how this led me to staticSearch. In this post, I describe my use of staticSearch, in hope that it might help others who, like me, want to include a seach engine in their online resource. I am using a Macintosh MacBook Pro, running Ventura 13.4 in July 2023 as I write this. (Revised September 2025, using Sequoia 15.7)
The documentation for staticSearch is at https://endings.uvic.ca/staticSearch/docs/index.html. The first step is to download the source code at https://github.com/projectEndings/staticSearch/releases/. This should arrive on your computer as a zip file named staticSearch-1.4.9.zip, or similar (September 2025). Just double-click to unpack the zip file into a folder named staticSearch-1.4.9. Move that folder somewhere convenient from the downloads folder: easiest and simplest to put it all in your Applications folder.
Before you can do anything more: you need Apache Ant. This is a tool designed to build complex software projects from source. Ant will read a set of instructions: get this file! rebuild it this way! save it to this file! now get another sofrware process to use that file to convert other files into something else! and then create new objects (new files, new tools, new libraries) from those files! etc. etc. Section 7.7 of the StaticSearch document says laconically:
Note: you will need Java and Apache Ant installed, as well as ant-contrib.
You should have Java already, in an up-to-date distribution, as part of your computer. But you may need to get Apache Ant. You get it from https://ant.apache.org/srcdownload.cgi. Look for the latest version: in September 2025, this was 1.10.15 (make sure you get the binary version). This requires Java 5, which you should already have. Download the zip file, double-click to unpack it to a folder named apache-ant-1.10.15 (or similar). As before, move that folder into your Applications folder.
You also have to get ant-contrib. This is a little more complex. What you actually need are two Java .jar files, named "cpptasks-1.0b5.jar" and "ant-contrib-1.0.jar". It took me a while to figure this out. The Apache ant-contrib page gives you the source for cpptasks, and someone with more expertise and time than me could (I suppose) compile the source into a Java .jar. But I took the short-cut and found a copy of cpptasks-1.0b5.jar out there on the net (in my case, at https://jar-download.com/artifacts/ant-contrib/cpptasks/1.0b5#google_vignette). I found ant-contrib-1.0.jar at http://www.java2s.com/Code/Jar/a/Downloadantcontrib10jar.htm.
Once you have these .jar files: place them both in the lib directory of your Apache Ant folder.
You are now ready to test out staticSearch. Here's what you do:
- Open the Macintosh terminal application. You will find this in your Applications/Utilities folder. This is a good old-fashioned command-prompt system, like we all used back in the 80s (remember the 80s? Wham? Freddy Mercury? yes, those).
- In the terminal: move into your static search folder. If you have unpacked it into Applications as "staticSearch-1.4.9" you should type "cd /Applications/staticSearch-1.4.9" into the terminal
- Now you are ready to test out all is working. For this you have to run Ant. You do this with the following command at the terminal:
/Applications/apache-ant-1.10.15/bin/./ant
(assuming you have got ant in a directory named apache-ant-1.10.15 inside Applications", and it is the binary version). - (September 2025). You may get an error " Only 1 operation can be specified". This appears to be the result of an error in the configTest.xml used by StaticSearch in this test. I have reported this to the makers of staticSearch.
- If you are smarter than I am you might be able to edit the $PATH statement in your terminal profile so that you just need to type "ant" into the terminal, and not "/Applications/apache-ant-1.10.15/bin/./ant". It seems Apple do not want you to edit your terminal profile, and are making this rather difficult: see https://stackoverflow.com/questions/9832770/where-is-the-default-terminal-path-located-on-mac.
<config xmlns="http://hcmc.uvic.ca/ns/staticSearch">
<params>
<searchFile>index.html</searchFile><recurse>true</recurse>
</params>
</config>
I now ran staticSearch on my material with this command:
/Applications/apache-ant-1.10.15/bin/./ant -DssConfigFile=mystuff/ssconfig.xml
This might fail because ant cannot find the key ssconfig.xml file. In that case, give the full path to the ssconfig.xml file:
/Applications/apache-ant-1.10.15/bin/./ant -DssConfigFile=/Applications/staticSearch-1.4.9/mystuff/ssconfig.xml
You may get an error "build failed", because a file "build.xml" can not be found. The build.xml file used by ant to create the staticSearch is in the /Applications/staticSearch-1.4.9 folder. If you are running ant from another directory you can find it by specifying the path to the build file by adding -f with the path to the build file:
/Applications/apache-ant-1.10.15/bin/./ant -DssConfigFile=/Applications/staticSearch-1.4.9/mystuff/ssconfig.xml -f /Applications/staticSearch-1.4.9/build.xml
The first time I tried this, it did not work. It turns out that the <params> declaration needs a whole lot more in it if you get a failed build. <params> needs to contain declarations as follows:
<phrasalSearch>true</phrasalSearch>
<wildcardSearch>true</wildcardSearch>
<createContexts>true</createContexts>
<resultsPerPage>5</resultsPerPage>
<minWordLength>2</minWordLength>
<maxKwicsToHarvest>5</maxKwicsToHarvest>
<maxKwicsToShow>5</maxKwicsToShow>
<totalKwicLength>15</totalKwicLength>
<kwicTruncateString>...</kwicTruncateString>
<verbose>false</verbose>
<stopwordsFile>test_stopwords.txt</stopwordsFile>
<dictionaryFile>english_words.txt</dictionaryFile>
<indentJSON>true</indentJSON>
It turns out that this issue is a part of a wider discussion in the SS community on what needs to be declared in the set-up, and what can be set as defaults. See the discussion in the comments on https://github.com/projectEndings/staticSearch/issues/270, where I first reported my experience, and on https://github.com/projectEndings/staticSearch/issues/195, where the wider discussion takes place.
These parameters require staticSearch to find the files test_stopwords.txt and english_words.txt. A copy of test_stopwords.txt is included in the test folder in the 1.4.9 distribution; a copy of english_words.txt is included in the xsl folder. Copy them into your mystuff folder, which should now contain:
Once you have successfully run ant staticSearch will create a new folder, called staticSearch, in this folder. The next step is to integrate the staticSearch into our own HTML. That's the subject of the next post.
The Endings Project and the Canterbury Tales Project (and also, Boccaccio and Dante)
At last, after many years, we (ie, me and a few other people) are getting ready to unleash on the world a whole series of digital scholarly editions. We have already released the second edition of Prue Shaw's Commedia, now at www.dantecommedia.it. We are now contemplating a third edition of that. Soon to come are Bill Coleman and Edvige Agostinelli's edition of Boccaccio's Teseida. And then the really big one: the first tranches of the Critical Edition of the Canterbury Tales. Based on All Known Pre-1500 Witnesses, with myself and Barbara Bordalejo as General Editors. All of these will appear in the next twelve months.
Why so long? We (as before) have been working on all these since the 1990s (the Dante and Chaucer) and 2000s (Boccaccio). There are multiple reasons. For this post, one reason is specially important: we wanted to be sure the edition could survive the chances of online time. It should stand alone, for decades and even centuries to come, as surely as a print edition might survive upon a library shelf. How could we achieve this, given all the shifting currents of the digital world?
We were not the only people worrying about this. From 2016 a five-year SSHRC grant (Canada) funded the Endings project. This project took as its starting point a number of digital projects based at the University of Victoria which faced exactly the same issue we had: how can these projects be given the best chance of survival long into the future? In fact, I did not come across the Endings project until a long way into the making of Shaw's second Commedia edition. By this time I had already reached identical (or nearly so) conclusions as the Endings project, as follows:
1. While our development of these editions had used custom database technologies to present and edit all project data, our published editions would not use databases or any related "server-side" technology at all: no databases, no PHP, no python, nothing. That is: everything would be contained on one server with no outside dependences at all so far as our texts are concerned
2. Our presentation of the texts would rely solely on the core web technologies of HTML5, css and javascript. Nothing else.
3. Any departures from these principles for any part of our edition (for example: the use of external JavaScript libraries; the use of IIIF image viewers) would use widely-used open source tools.
These principles correspond the Endings project principles 4.1, 4.2 and 4.9. In some areas, however, our practice differs from that of the Endings project. For example, we do use the JQuery library, which in my view has now achieved core web technology status. I think the same is becoming true of the IIIF family. However, I do not think the same is true of XML technologies (nor, interestingly, do the Endings people) and we do not use XSLT, etc, as any part of our final publication model. We also use query strings, which again seem to me a core web technology, where Endings does not. Nor do we aim for "graceful failure" where css/javascript/something else does not work. It seems to me that providing all source data within the edition, permitting others to fashion new interfaces to our data, is the best way of anticipating any failure.
One might object: we are making a bet on certain core technologies now still being core technologies centuries in the future. Yes we are. But we see this bet as being in the same category as the bet scholars have made for millennia: that there will be a library or other place somewhere in the future which has a shelf for my book.
Another principle of the Endings project is that it will not use an external service to provide functionality, and specifically names Google Search as such a service. In my early preparations for the Shaw edition, I had investigated using Google Search to provide a search tool. Indeed, the second edition at www.dantecommedia.it implements searching in exactly this way. You can see from just a cursory use of Google Search in the second edition how unsatisfactory it is. Searching for "come", one of the most common words in the Commedia, gives just one result; "tanto" yields none at all. Many search results begin with advertisements, for holidays, or beer. I spent many hours trying to get Google Search to do better, including feeding it hard-wired urls to every page of transcription. Nothing seemed to work. It appears the Google algorithms rebel when faced with nine near-identical texts, and fail over and over to return anything like meaningful results.
For these reasons I was contemplating just how a stand-alone search system might be implemented, when I came across the Endings project, and StaticSearch. They had done it! and it worked! On another page, I describe my experiences of StaticSearch.
Tuesday, 20 July 2021
Fun with Fonts. Junicode, Unicode, and ꝑ
If you see a character looking like a p with a bar through the descender in the title of this post, and you see it here too ꝑ, then ... read on. And if you don't, then read on (and let me know!)
Thirty years ago, when myself, Tim Berners-Lee, Lou Burnard and the web were much younger, every "special character" was a challenge, and a potential triumph or failure. "Special" meant something beyond ASCI 127 (ah, the acronyms!). It meant anything non-English, in the most limited BREXIT sense. E-acute was used by people from across the Channel, and a few Canadians, and not to be used without Special Equipment (in those days, a Macintosh computer). Devanagari was a distant dream, and right-to-left writing, an impossibility.
Nowadays, thanks to Unicode, and the work of many unsung heroes of font-design, with a special shout-out to those who sat on myriad committees and shepherded the whole process to every smart phone on the planet, we have become so used to everything appearing just right, with no effort at all on our part, that we are in danger of forgetting how many miracles had to occur so that I can insert a ꝑ in my document, and you can see it. (The best miracles are made by people working together, of course). But every now and then, something happens to remind us of how many ducks make a row.
Like many medievalists, I am a fan of Peter Baker's beautiful Junicode font. For years, I have been happily typing ꝑ into transcriptions, Word and pdf documents. This and a few other characters are very common in many medieval vernacular and Latin manuscripts. ꝑ is used as an abbreviation for per or par, as in "person" and "parish", and so found everywhere in Chaucer manuscripts (think of the Parson and the Pardoner). One of the great joys of Junicode is that it shows this character in a particularly elegant form, appearing as
Over the years, we have used Junicode in all our work with medieval texts, and have become so accustomed to the daily miracle of Junicode that we don't think about it. It works. "We" is all the people who work on the Canterbury Tales Project and a few other projects -- particularly Dante. I am currently working with various Dante scholars on a new publication, coming soon to a browser near you. Trust me, you will know about this when it happens. So, imagine my surprise when after so many years of trouble-free use, my main collaborator said that our elegant Junicode p with bar appeared as a horrid oversize black character on her computer, thus:
At first, I thought this was just an aberration, something odd about the way her computer was set up. The character appeared fine on my computer, and on various other computers I looked at, but not on hers. Why not? Down the rabbithole I went.
By this time, we had graduated to bundling the Junicode font with our developing site, so that readers would not have to download the font to their computer. This a well-documented process, and Squirrel font documents it and provides neat tools to convert any font to a "webfont", easily embeddable in any web page. So I began investigating. On my computer, the character appeared fine:
- if I had Junicode on my computer, and the font embedded in the page
- if I had Junicode on my computer, and the font NOT embedded in the page
It did NOT appear fine if I did NOT have Junicode on my computer and had only Junicode embedded in the web page. Yet the web page showed Junicode everywhere else -- but not this character, and a few other characters. How could this be?
I began digging. The unicode code point for p with a bar is A751. This is in the "general use" area of unicode, which major fonts will support as a matter of course: so you can paste the ꝑ from this document into a Word document and use it in Times New Roman, Geneva, etc. When I looked at Junicode in my computer, using Apple's Font Book, p with a bar appeared as glyph 2007, Unicode A751, exactly as it should:
However, on my collaborator's computer, the same character appeared in a quite different place: as glyph 2066, unicode E670 (on my computer, Junicode has a quite different character at glyph 2007).
What is going on? Why is her Junicode different from mine? On digging about, it appears that some time in the past, Junicode indeed had this character at E670. The "E" and "F" unicode ranges are "Private Use" areas, and it appears that up to the time when p with a bar was allocated A751 in the "general use" area, Junicode put p with a bar in the "private use" area, with that encoding. This is a rather long story, involving a group called the Medieval Unicode Font Initiative (MUFI). One of the aims of this group was to have "core" characters judged as essential to scholars working with medieval western European texts incorporated into the "official" Unicode encoding. As of Unicode 5.1, 152 MUFI characters -- among them, p with a bar -- had made it into official unicode. It appears that my version of Junicode reflects this shift of p with a bar into official, post 5.1, unicode. The version of Junicode on Prue's computer did not.
More digging. By this time, I was suspecting that the embeddable version of Junicode did not have p with a bar at A751. But why did it display correctly on my computer? It appears that somewhere deep in the innards was an instruction to the effect: if the browser could not find the character in the embedded font, look elsewhere: so it looked in the Junicode on my computer, found it and displayed it. It did this even when I tried to fool it by calling the embedded font something else in the CSS ("junicoderegular") style sheet. However, on my collaborator's computer the character did not appear as A751, and so it showed an A751 from another font altogether.
Eventually, after scores of emails and hours of digging, I concluded that the root of the problem lay in the embedded font. Somehow, this embedded Junicode did not have p bar where it should be. So I set to trying to correct this. First I went to the Squirrel font generator:
I uploaded the Junicode TTF from my computer, Squirrel converted it to a "webfont", and all seemed fine. Nope. Same problem. I dug deeper. I went to Peter Baker's "Junicode" page on FontSquirrel and used the "webfont kit" generator on that page. Nope. Same problem. With increasing desperation, I noticed that the page offered a choice of "subsets":
So, I chose "no subsetting" and created the webfont. And at last! it worked!
All this for characters which appear just five times in some 2400 pages of manuscript transcription.
This tale casts into relief the many rough edges that exist in the interplay of fonts, glyphs, character coding points, unicode spaces, and encoding systems (utf8? or 16? BOM or not?), all playing against multiple versions as all of these evolve and agreements are forged and renewed. The wonder is that problems like these occur so rarely.
Tuesday, 13 February 2018
Getting Started with Textual Communities
Welcome to the temporary home of Version 2 of Textual Communities ("TC"), at textcomtest.usask.ca. This address will change when we are ready to go fully public with TC. Until then, this is a sandbox version, and all data may disappear at any point.
If you just want to see what TC can do: choose a community from "Public Communities", and "View".
<?xml version="1.0" ?>
<TEI xmlns="http://www.tei-c.org/ns/1.0">
<teiHeader>
<fileDesc>
<titleStmt><title>Fairfax</title></titleStmt>
<publicationStmt>
<p>Draft for Textual Communities site (spelling modernized)</p>
</publicationStmt>
<sourceDesc><p>Murray McGillivray</p></sourceDesc>
</fileDesc>
</teiHeader>
<text>
<body>
<pb n="130r" facs="FF130R.JPG"/>
<div n="Book of the Duchess">
<lb/><head n="Title">The book of the Duchesse</head>
<lb/><l n="1">I Have great wonder/ be this light</l>
<lb/><l n="2">How that I live/ for day nor night</l>
<lb/><l n="3">I may nat slepe/ wel nigh nought</l>
<lb/><l n="4">I have so many/ an idel thought</l>
<lb/><l n="5">Purely/ for default of sleep</l>
<lb/><l n="6">That by my truthe/ I take no keep</l>
<lb/><l n="7">Of no thing/ how it cometh or goth</l>
<lb/><l n="8">Ne me is no thing/ leief nor loth</l>
<lb/><l n="9">Al is y like good / to me</l>
<lb/><l n="10">Joy or sorrow / where so it be</l>
</div>
</body>
</text>
</TEI>
There are a few things to note about this file:
If you just want to see what TC can do: choose a community from "Public Communities", and "View".
Sample files
You can get the sample files used in this documentation at www.sd-editions.com/tc. You can download all the files in this directory in a single zipfile at www.sd-editions.com/tc/tcstart.zip.
Logging in
Here is what you see:
Press the inviting "Start" button, and you will be asked to log in by social media, or create a log-in using your email address. If you do the latter, you will be sent an email to that address to confirm your registration. (Note: TC uses email addresses to uniquely identify each user).
Creating or joining a community
When you first log in as a new user, the Start button has changed:
The "Create Community" button brings you to this screen:
The two compulsory fields, "Name" and "Abbreviation", are marked with *. Note the accessibility options: you can hide your community from everyone, or allow anyone to do anything, and many options in between.
Your first document: an XML file
Once you have a community, you need documents! The "Start" button at the centre of the screen has changed again:
Choose "Add Document" and you are offered two choices:
This time, select the "XML file" option. TC likes TEI! Here is a very simple example of a TEI/XML file, optimized for TC use:
<?xml version="1.0" ?>
<TEI xmlns="http://www.tei-c.org/ns/1.0">
<teiHeader>
<fileDesc>
<titleStmt><title>Fairfax</title></titleStmt>
<publicationStmt>
<p>Draft for Textual Communities site (spelling modernized)</p>
</publicationStmt>
<sourceDesc><p>Murray McGillivray</p></sourceDesc>
</fileDesc>
</teiHeader>
<text>
<body>
<pb n="130r" facs="FF130R.JPG"/>
<div n="Book of the Duchess">
<lb/><head n="Title">The book of the Duchesse</head>
<lb/><l n="1">I Have great wonder/ be this light</l>
<lb/><l n="2">How that I live/ for day nor night</l>
<lb/><l n="3">I may nat slepe/ wel nigh nought</l>
<lb/><l n="4">I have so many/ an idel thought</l>
<lb/><l n="5">Purely/ for default of sleep</l>
<lb/><l n="6">That by my truthe/ I take no keep</l>
<lb/><l n="7">Of no thing/ how it cometh or goth</l>
<lb/><l n="8">Ne me is no thing/ leief nor loth</l>
<lb/><l n="9">Al is y like good / to me</l>
<lb/><l n="10">Joy or sorrow / where so it be</l>
</div>
</body>
</text>
</TEI>
There are a few things to note about this file:
- "Content" elements with "n" attributes (<l n="1">) are especially important to TC. TC uses these to identify all content sections. Thus: the first line is labelled by TC as "div=Book of the Duchess:l=1", and TC then uses this identifier to locate all versions of the first line in every document
- Note the explicit use of <lb/> elements to mark each document new line. TC uses the implicit hierarchy of page, column and line breaks (<pb/> <cb/> <lb/>) to construct a "text-tree" for each document, alongside the "text-tree" it creates for the hierarchy of <div> and <l> elements.
Adding more documents, adding images
After selecting "XML file" you will get this dialogue:
Choose the file "Fairfax.xml" from the sample files (see above), give it the name "Ff" (or similar), and press "Load".
You will receive various encouraging messages, and the window should change to show you the sigil for this manuscript in the left hand pane:
Click on the arrow beside Ff to see the pages in Ff, and then click on the first page. Its transcription will now appear in bottom right pane:
Now, you can add an image to the page. You can do this in several ways:
- Click on the "Add Image" button in the top-right pane, or the camera icon beside the page number "130r". You will get a box inviting you to choose an image file or drop it onto the dialogue. Choose FF130R.JPG from the sample files.
- You can load multiple images by putting them all in a folder, zipping the folder, and then clicking on the ZIP icon next to the manuscript name. Choose FairfaxImages.zip from the sample files.
Play around with the other icons on this page. Try pressing the "Save" "Preview" and "Commit" buttons, to see what happens. (Note: "Commit" will write the page to the underlying database.)
Add another document by clicking on the + icon in the left hand pane. Again, choose the "XML file" option, this time add "Bodley.xml" from the sample files, with the name Bd.
Add another document by clicking on the + icon in the left hand pane. Again, choose the "XML file" option, this time add "Bodley.xml" from the sample files, with the name Bd.
Collation
The power of Textual Communities may be seen in the Collation system. At the top of the left panel, click the "Collation" tab:
In TC terms, an "entity" is a discrete segment of an act of communication: a line of poetry, a paragraph of prose. Click on the arrow beside "Book of the Duchess" to open up the entities (lines of poetry) within it:
(The order of these may vary.) Now, click on one of these lines. You will get this advice:
So, go to that menu:
Choose a base text (it does not matter which). Now, go back to click on line 1 in the collation. The right hand panel will change, to present the wonderful Collation Editor (developed originally for the Greek New Testament editing projects at Münster and Birmingham):
(You may need to make the window larger to see the menu at the bottom of the pane). Spend some time playing with this. You can regularize variants (e.g. remove the variant wonder/wondir) by dropping one word on another:
After choosing "Save", you will see that both manuscripts now have the reading "wonder":
Play with the settings menu. You can change how the collation works from this menu:
You will see how the collation changes as these selections change.
This brief introduction gives only a glimpse of the power of the Collation Editor. Try the following, for example:
Then, commit this change and return to the collation. You will see that line 38 now includes this extra text, across the page break. You can view the XML for this page by clicking on the XML icon beside the manuscript name, to comfirm that the line indeed continues across the page break:
This brief introduction gives only a glimpse of the power of the Collation Editor. Try the following, for example:
- Go back to one of the documents, change line 1, commit the change (this writes it to the database used by the collation), and return to the collation. You will see your change there.
- Now, for fun: go to the second page of Ff (130v) and have line 38 continue from the previous page onto this page and add something to it. Hint: change the "From previous page" value:
Other facilities
There is a great deal more in TC than this sketch shows. It is particularly rich in community management features, as follows:
- You can invite other people to become members of your community (click on the "Members" link when you have chosen your community, or on the "Member profile" item on the log-in menu) and follow the "Invite" link
- You can change the status of any member, assign them pages to transcribe, check the progress of the transcription, assign them someone to approve their transcripts (the "Members" link for each community you lead)
- You can permit other people to join your community without need of your approval, or require that anyone who wants to join must be approved by you ("Member profile" on the log-in menu)
Copyright, etc.
We encourage anyone contributing materials to TC to make these available under the Creative Commons Attribution (CC-A) license. That is: no share-alike and no "non-commercial" restrictions. This means there should no restrictions at all except requiring all subsequent users of the material to acknowledge your part in making it.
For the time being: TC will accept materials which do have restrictions on them. However, it is likely that TC in future will require that all materials held on TC servers are free of all restrictions (CC-A or similar). This is because TC uses University of Saskatchewan and Compute Canada servers. As both are publicly funded, hosting materials with any kind of copyright restrictions raises legal and ethical issues.
If this is a problem for you, you should not use TC.
Some interesting features of TC
Here, in no particular order, are some aspects of TC which make it unusual, even unique:
- TC is built on an explicit ontology of texts, documents and works. Various of my publications describe this ontology (see https://www.academia.edu/12297061/Some_principles_for_the_making_of_collaborative_scholarly_editions_in_digital_form; https://www.academia.edu/9575974/The_Concept_of_the_Work_in_the_Digital_Age_published_version_; https://www.academia.edu/3233227/Towards_a_Theory_of_Digital_Editions). Briefly: TC sees text as a collection of leaves, with all leaves present on two distinct trees, each of which conforms precisely to the "OHCO" (ordered hierarchy of content objects) model. One of the trees represents the document (codex/quires/pages/columns/lines). The other tree represents the act of communication ("entity") inscribed in the document: as Play/Scenes/Acts/Lines, or Poem/Stanzas/Lines, etc. Note that this is not simply a matter of "overlapping hierarchies", as usually characterized. It is actually two quite distinct trees: distinct to the point that branches and their leaves might appear with quite different orders on the two trees (as in the case of notes or alterations spanning across the margins of multiple pages, etc.) Broadly, TC uses the 'document' tree to display the document page by page, line by line, and TC uses the 'entity' tree to locate units of text across multiple documents for collation.
- XML and all the tools associated with it famously supports "one text, one tree". (Long ago, XML's predecessor SGML did attempt to enable multiple trees in any one text through the CONCUR feature. I never did discover a useful implementation of CONCUR.) Over some twenty-five years, I have tried to manipulate the two hierarchies using a variety of tools (most prominently, the Anastasia publishing system). One problem was that for long I thought the problem was simply "overlapping hierarchies", and not the more demanding scenario of two distinct trees. Another problem was the inefficiency of XML tools. Accordingly, while TC uses XML as its standard input format, it creates the two distinct trees from the XML and then stores the two trees not as XML but as a series of JSON documents stored in a MongoDB backend. In essence, the text is a collection of leaves stored in JSON fields, with each leaf also stored in distinct JSON documents representing the two trees. Over the last decade I have attempted to express this model with three different database systems: first, XML in the form of XML-DB; then SQL in a relational database (underlying the first version of TC, still to be seen at www.textualcommunities.usask.ca), and finally JSON. JSON wins. A key reason for the success of JSON was the requirement that we be able to edit pages in real time: that is, take out a chunk of each tree, rebuild both trees as needed and then reattach the leaves of text to each rebuilt tree, all while the editor watches. Doing this in real time is like gathering leaves in a howling gale. As a bonus, JSON (much more than XML) is the native language of web content, with an immense range of Javascript/HTML tools available to process it.
- Technically: TC is build in pure javascript, using node.js and npm tools (https://nodejs.org/en/; https://www.npmjs.com/), for both server and browser components. This makes maintenance, etc, far easier. TC also uses the Angular framework to provide all interface components (https://angularjs.org/; drawing on the Bootstrap and JQuery libraries). This architecture was designed by Xiaohan Zhang between 2012 (when we realized that the SQL solution would not work) and 2015. All code is freely available on Github, at https://github.com/DigitalResearchCentre/tc.
- Theoretically: there is no limit to the number of trees structuring every text. TC supports two. Best of British luck to whoever wants to deal with more than two.
- TC uses a IIIF server and viewer software (http://iiif.io/). In the future, we want to broaden our support for IIIF, to import full IIIF documents, etc.
- We would like to be obsolete very very soon. Someone please do this better than we did.
Subscribe to:
Posts (Atom)