------------------------------------- | ScrollKeeper 0.2 Specifications | | | | Laszlo Kovacs & Dan Mueth | | | | March 29, 2001 | ------------------------------------- This document is arranged as follows: I. Introduction II. Definitions and Overview III. Building, Installing, and Uninstalling IV. Other Tools for Admin and Packaging V . ScrollKeeper Internals VI. Example File Snippits I. Introduction This document describes the functional design of ScrollKeeper and the basics of how ScrollKeeper can be used in a self-consistent way. For more details on details about the implementation, read the source or ask on the mailing list. For more information on how to use ScrollKeeper, see the example package, scrollkeeper-example1, and the documentation that accompanies it. Note: Several parts of ScrollKeeper were designed around the needs of packaging systems such as RPM. In this document RPM is used as an example packaging system, although other packaging systems should be treated similarly. ScrollKeeper is intended to work with various packaging system on Unix-like operating systems. II. Definitions and Overview Contents List - This is a list of documents on the system (or more generally, including docs which are not local to the single machine), sorted into a tree. The tree is usually sorted by subject using a controlled list of subjects. TOC (Table of Contents) - This is a tree representing the sections and subsections of a document. The TOC can be extracted from an SGML/XML file. Extended Contents List - This is the Contents List with the TOC for each document merged into it. It allow the user to not only navigate through the the docs on the system, but also the sections within each doc, all in one tree. Generally a help browser will want to present all of the above information to the user. The Contents List is determined from all of the OMF metadata, which is obtained either from standalone OMF files or (in the future) by extracting it directly from DocBook documents which have this metadata in them. The TOC is generated directly from DocBook documents. Since all of this is metadata and must be accessed rather frequently, and to avoid potential performance problems, this metadata is processed only at install and uninstall time. The data is stored in XML files for easy access by the help browser and ScrollKeeper as necessary. Notes regarding ScrollKeeper 0.2: * ScrollKeeper 0.2 only supports local documents. * ScrollKeeper 0.2 uses a subject categorization and a strictly-enforced controlled list for the categories. * ScrollKeeper 0.2 extracts the TOC from DocBook/SGML documents. III. Building, Installing, and Uninstalling i. Building The OMF documents are manipulated during the application build process, generally as part of 'make all'. The purpose of this is to specify the URL of the document where it will actually be installed, which is not generally known until build time. This processing is done by 'scrollkeeper-preinstall'. This generally takes three parameters: the URL for the installed document, the original OMF metadata file in XML, and the name that the generated OMF file should be written to. For example: scrollkeeper-preinstall \ file:$(docdir)/foo-manual.sgml \ foo-manual-fr.omf \ $(omf-dir)/foo-manual-fr.omf Note: Note that this requires ScrollKeeper to be installed on the machine of the packager. Note: Also note that $(docdir) typically is based on $(prefix) which is set to include $RPMBUILDROOT during the 'make install' step when building RPMs. Thus, one must use timestamps to guarantee that the 'scrollkeeper-preinstall' step is only executed once, during the 'make' step and not during the 'make install' step. Otherwise the URL will be wrong, having the $RPMBUILDROOT in it. If the path where the generated OMF file is written does not exist, it is created. So in this example, the OMF file $(omf-dir)/foo-manual-fr.omf is created by copying the OMF file foo-manual-fr.omf, replacing the IDENTIFIER with file:$(docdir)/foo-manual.sgml. If $(omf-dir) did not exist, it is created. Note that the OMF file is named -.omf to avoid collisions between OMF files within a package when they are installed. ii. Installing Installation is done by first copying the document and OMF file into place and then having ScrollKeeper update its database by calling 'scrollkeeper-update'. install: install foo-manual.sgml $(docdir) install $(omf-dir)/foo-manual-fr.omf $(pkgomfdir) -scrollkeeper-update -p $(SCROLLKEEPER_DB_DIR) SCROLLKEEPER_DB_DIR is the directory to use as the ScrollKeeper database directory. It should be based on $localstatedir so that it points to the real system database when installing, and point the RPMBUILDROOT when building RPMs. For example, if the application is configured with --localstatedir=/var/lib, then SCROLLKEEPER_DB_DIR should be set to $(localstatedir)/scrollkeeper. The -p option is important for building and testing purposes and should always be used when packaging applications. When building RPMs, for example, it guarantees that instead of actually modifying the primary database on the builder's machine, it generates and modifies a new database under RPMBUILDROOT. It is clearly useful for testing purposes as well. scrollkeeper-update identifies the presence of a new or modified OMF file in $(omf-dir)and registers new metadata. Uninstalling is done using 'scrollkeeper-update' as well. uninstall: rm $(docdir)/foo-manual.sgml rm $(pkgomfdir)/foo-manual-fr.omf scrollkeeper-update -p $(SCROLLKEEPER_DB_DIR) scrollkeeper-update identifies that the OMF file was removed from OMFDIR and unregisters the old metadata. scrollkeeper-update also has a "-v" (verbose) flag which sends warnings and errors to STDOUT in addition to the log file (/var/log/scrollkeeper.log) where these messages are normally sent. IV. Other Tools for Admin and Packaging In addition to the commands described above (scrollkeeper-preinstall and scrollkeeper-update), ScrollKeeper provides the following additional utilities: scrollkeeper-rebuilddb: This is used to rebuild the ScrollKeeper database completely using only the original OMF files in OMFDIR. It also has a "-p" option which allows one to specify the output directory of the newly created database. It has a "-v" (verbose) option to output errors and warnings to the STDOUT. All errors and warnings are also sent to the log file (typically /var/log/scrollkeeper.log). scrollkeeper-config: This is used to return various configuration paths for ScrollKeeper. Options: --help Prints scrollkeeper-config usage. --version Prints Scrollkeeper version. --prefix Scrollkeeper configure prefix. (eg. /usr) --localstatedir Scrollkeeper configure localstatedir. (eg. /var) --pkglocalstatedir Scrollkeeper data directory. (eg. /var/lib/scrollkeeper) --pkgdatadir Scrollkeeper home directory. (eg. /usr/share/scrollkeeper) --omfdir Directory where ScrollKeeper looks for OMF files. (eg. /usr/share`/omf) V. ScrollKeeper Internals This section describes some details about scrollkeeper functions. $SCROLLKEEPER_DB_DIR is the directory where ScrollKeeper keeps its current data files. (It is typically /var/lib/scrollkeeper.) $SCROLLKEEPER_DIR is the directory where ScrollKeeper installs its static data files and docs (typically /usr/share/scrollkeeper). All the content list template files for every locale are stored here. $SCROLLKEEPER_DOC_DIR is the directory where ScrollKeeper installs its development related info files such as README, license files etc (typically /usr/share/doc/scrollkeeper-$version). $OMFDIR is the directory where packages must install their OMF files and where ScrollKeeper looks to find OMF files. (This is typically /usr/share/omf.) scrollkeeper-update [-v] [-p ] [-o ] It looks for added, removed, or updated files in OMF_DIR and its subdirectories and then calls scrollkeeper-install and scrollkeeper-uninstall as necessary. -v - verbose flag, messages are printed to stderr also apart from /var/log/scrollkeeper.log. -p - specify the Scrollkeeper database directory (defaults to scrollkeeper-config --pkglocalstatedir) -o - specify an OMF directory (defaults to scrollkeeper-config --omfdir) Multiple OMF directories can be specified by setting the environment variable OMF_DIR. scrollkeeper-install [-v] [-p ] (If the -v flag is used, various errors will be sent to STDERR throughout the following process.) 1) Define a path, $scrollkeeper_db_dir, to be if the -p option is used, or `scrollkeeper-config --pkglocalstatedir` otherwise. 2) If the content list template files were not all copied to the database directory (first run) then copy them and create the extended content list files also (right now these will be the same as the simple content list files). 3) For each entry in do the following: A) Verify that the URL in the OMF entry exists. (eg. the file it points to on the drive actually exists.) If it fails this test, abort all further action on this OMF entry. B) Add the entry to $scrollkeeper_db_dir/scrollkeeper-docs. (This file is created if it doesn`t exist.) (see V. for format) C) Update Contents List: a) Determine title, language, and position(s) in Contents List from TITLE, LANGUAGE, and SUBJECT.category respectively. b) Verify that the SUBJECT.category lies within the controlled list of categories for the locale specified by LANGUAGE c) Add this document to the Content List file in the appropriate language's directory: $scrollkeeper_db_dir//scrollkeeper_cl.xml (See V. for format.) (Note the doc ID is used as an attribute to make uninstallation easy.) D) (For DocBook docs only.) Extract TOC from doc, placing it into a file: /TOC/ (This file is created if it doesn`t exist, or overwritten if it does.) E) (For SGML and XML docs only.) Insert TOC into an "extended contents list". This is the Contents List with TOC listed as a sub-tree for each doc. As above, the doc ID is used to make uninstallation easy. This file is at: $scrollkeeper_db_dir//scrollkeeper_extended_cl.xml scrollkeeper-uninstall [-v] [-p ] (Flags are the same as those in scrollkeeper-install.) 1) Define a path, $scrollkeeper_db_dir, to be if the -p option is used, or `scrollkeeper-config --pkglocalstatedir` otherwise. 2) Identify the doc ID of the specified doc using $scrollkeeper-db-dir/scrollkeeper_docs 3) Remove the doc from $scrollkeeper-db-dir/scrollkeeper_docs and $scrollkeeper-db-dir//* (both contents list files). 4) Remove the TOC file for the doc: $scrollkeeper-db-dir/TOC/ scrollkeeper-rebuilddb [-v] [-p ]: (Flags are the same as those in scrollkeeper-install.) 1) Define a path, $scrollkeeper_db_dir, to be if the -p option is used, or `scrollkeeper-config --pkglocalstatedir` otherwise. 2) Delete the old database (ie. 'rm -rf $scrollkeeper_db_dir/*') 3) Run 'scrollkeeper-update -p $scrollkeeper_db_dir' which creates a new database in $scrollkeeper_db_dir and registers all OMF files in it. (This can be used whenever the database is corrupted or becomes out-of-date. It is also an easy (brute-force) way to allow for ScrollKeeper to update its databases when a new version is installed, allowing one to break database compatibility between versions of ScrollKeeper.) -- For use by help browsers -- scrollkeeper-get-contents-list : 1) Returns the path: ${pkglocalstatedir}//scrollkeeper_cl.xml if it exists scrollkeeper-get-extended-contents-list : 1) Returns the path: ${pkglocalstatedir}//scrollkeeper_extended_cl.xml if it exists scrollkeeper-get-toc-from-docpath : 1) Looks in ${pkglocalstatedir}/scrollkeeper-docs to find doc id for 2) Returns the path: ${pkglocalstatedir}/TOC/ VI. Example File Snippits scrollkeeper-docs: This is used to keep track of which docs are on the system listing the location of their OMF metadata, and identifying each doc by a code which can be used to simplify uninstallation and some other tasks. It also has a timestamp from when the OMF file was last written so that scrollkeeper-update can detect when an OMF file has been updated. Finally it also holds the locale to which the doc was installed. /usr/doc/omf/foo-manual-fr.omf \ 000001 \ /usr/doc/foo/foo-manual.sgml \ Sun Oct 29 13:48:50 \ C /usr/doc/omf/bar-manual-en_US.omf \ 000002 \ /usr/doc/bar/bar-manual.sgml \ Tue Oct 24 02:47:28 \ en_US /usr/doc/omf/foob-manual-da.omf \ 000003 \ /usr/doc/foob/foob-manual.sgml \ Tue Oct 24 02:47:28 \ da scrollkeeper_cl.xml: This is a simple XML file which reveals what the Contents List looks like and where the OMF files and doc are at. This is used by the help browser to display the Contents List. [We will need to write up a simple DTD.] Applications Games FreeCell /usr/local/games/freecell/freecell-omf.xml /usr/local/games/freecell/freecell.sgml SGML TOC/: Contains a description of the sectioning layout of the doc. It also provides URI's for each doc section. Sections without URIs are not included, since they can't be linked to. scrollkeeper_extended_cl.xml: Similar to scrollkeeper-contents-list.xml, except has TOC info in it too. This is used by the help browser to display the "Extended Contents List". Applications Games FreeCell /usr/local/games/freecell/freecell-omf.xml /usr/local/games/freecell/freecell.sgml SGML Introduction to \ FreeCell Playing FreeCell Winning FreeCell foo-manual.omf: This is the metadata file of the doc. Typically it looks like: ScrollKeeper Manual Dan Mueth d-mueth@uchicago.edu System|Other This document describes what ScrollKeeper is, the basics of how it works, basic system administration and maintenance of ScrollKeeper, how to create packages which register documents with ScrollKeeper, and the exported functionality for help browsers. manual It is based on the OMF specification about document metadata: http://www.ibiblio.org/osrt/omf/