Various Documentation

Universal Package Registry

Universal Feeds and Packages are "lightweight" and, on their own, have very few built-in features. This design has allowed them to be utilized for all sorts of packaging problems such as application delivery, Inedo's product extensions, and even private Bower packages.

When developing a package-based solution, one question that often comes up is "what packages are installed or used in this particular context?" That's where the Universal Package Registry comes in; it's a local package registry designed specifically for Universal Packages.

Background: Other Local Package Registries

All packages, universal format or otherwise, are essentially zip files with content, (i.e. the actual files you want packaged) and a metadata file that describes the content. Once the contents of a package are extracted and installed to a directory, there’s no easy way to know which package those contents came from, or if they came from a package at all. This is where a local package registry comes in.

Different package managers use different mechanisms to represent a local registry. Some store the entire package, others store only metadata about the package, and each approach has advantages and disadvantages.

For example:

  • The local registry for MSI (Microsoft Installer) is comprised of an installation cache directory (containing the MSI files installed) as well as various entries in the Windows registry.
  • The NuGet local registry is comprised of various package cache directories and a packages.config file stored in the root of a project file that describe which packages are used.
  • APT (Advanced Package Tool for Debian) maintains a directory of package-related files in /var/lib/dpkg/info, including MD5sums of each of various installed files and the executable scripts that would be run before and after installation.

However, in all cases, the packages registered vs packages installed are not enforced by the package manager or operating system. You can consolidate the data in the local registry if needed, or delete files that were originally installed by the package manager.

Designing the Universal Package Registry

The Universal Package Registry was designed with the realities of other local package registries in mind.

  • Well-defined specification - this allows a variety of tools (not just an API or CLI) to read or write data
  • Data only - there is no attempt to automatically reconcile registered vs installed packages
  • Compliance-driven metadata - provides a common mechanism to indicate what installed a package and why
  • Any additional metadata - for compliance or other reasons, you can add additional package metadata

A Universal Package Registry may also contain a package cache. There are three common uses cases for this:

  • Auditing - to verify or compare the original contents of the package registered versus the files installed on disk
  • Repairing installation - in the event an installed package was corrupted (a file in the install location was modified or deleted), the package could be reinstalled without needing to connect to a universal feed
  • Staging packages - packages files could be copied to the cache ahead of installation to allow for rapid deployment of newer versions

Universal Package Registry Specification

A Universal Package Registry has two components: a registry file and ephemeral lock file. A package cache can also be added but it's optional. Its layout on disk is as follows:

‹registry-root›\ .lock installedPackages.json packageCache\ ‹group$packageName›\ ‹packageName.version›.upack

Like with universal packages, you can add any number of files or directories outside of these minimal requirements. However, we strongly recommend that you prefix these files and folders with an underscore (_) as not to clash with files or folders that are added in a future version of the specification.

Interacting with a Universal Package Registry and the Lock File

The lack of a registry root directory or installedPackages.json file is not an error condition, but implies that no packages are registered (i.e. have been installed). An invalid installedPackages.json file (i.e. not readable as JSON or invalid data) is an error condition and should not be automatically remediated.

The .lock file is used to indicate that another process is currently interacting with the registry. It should only be used when atomic reading/writing of the metadata file; modifying the package cache should not cause the repository to be locked.

When a .lock file exists, its modification date should be checked against the current system time. If the difference is greater than ten seconds, the other process is assumed to have crashed and the lock file should be deleted. Otherwise, the file should be rechecked in this manner until the lock is freed.

When no .lock file exists, a process should create a lock file with two lines (\r or \r\n): a human-readable description of the lock (generally the process name), and a lock token (generally a GUID). If, at the completion of the operation, the lock token matches, then file should be deleted.

No operation should take more than a second (let alone ten), and the user should be notified of all exceptions (locked registry, mismatched token).

Universal Package Registry Files

The registry file (installedPackages.json) is a JSON-based array of objects with the following properties.

Property Format
group see package metadata specs
nameR see package metadata specs
versionR see package metadata specs
path A string of the absolute path on disk where the package was installed to
feedUrl A string of an absolute url of the universal feed where the package was installed from
installationDate A string representing the UTC date when the package was installed, in ISO 8601 format (yyyy-MM-ddThh:mm:ss)
installationReason A string describing the reason or purpose of the installation

For example, BuildMaster uses {Application Name} v{Release Number} #{Package Number} (ID{Execution-Number})
installationUsing A string describing the mechanism the package was installed with; there are no format restrictions, but we recommend treating it like a User Agent string and including the tool name and version

For example, BuildMaster uses BuildMaster/5.6.11
installationBy A string describing the person or service that performed the installation

For example, BuildMaster uses the user who triggered the deployment or SYSTEM if it was a triggered/scheduled deployment

An R denotes a required property, and the object may contain additional properties as needed.

We strongly recommended that you prefix these properties with an underscore (_) as to not clash with property names that may be added to the specification later.

Package Uniqueness and Data Constraints

Only one version of a package may be registered at a time. Uniqueness is determined by a combination of the group (or lack of a group) and package name. A future version of this specification may allow for multiple versions of a package, but that will be an “opt-in” setting likely defined in a (to be specified) registry configuration file.

Package Cache

The package cache is simply a directory containing package files that may currently be installed. It must be named packageCache, and contain package files (packageName.version.upack) stored in subdirectories comprised of the group name (with $ replacing the /), a $, and the package name. For example:

‹registry-root›\ packageCache\ $hdars\ hdars.1.2.3.upack hdars.1.2.4.upack hdars.2.0.0.upack accounting$apps$accounts\ accounts.1.0.2-beta.upack

We strongly recommend that cache usage to be explicitly opt-in (both when downloading and installing packages). Default package caching generally causes more problems than it helps, and can occupy a lot of disk space if not monitored.

Registry Types and Locations

There are three different types of registries (Machine, User, Custom). Each Machine can have any number of User and Custom registries:

  • Machine-level Registry - a registry with a well-known location that is queryable by anyone and, by default, requires administrative privileges to install/change
    • Windows: %ProgramData%\upack
    • Linux: /var/lib/upack
  • User-level Registry - a registry with a well-known location based on the current username that can be changed by the current user and administrators
    • Windows: %USER%\.upack
    • Linux: ~/.upack
  • Custom registry - a location with a different location and potentially different privileges

Implementation: upack.exe

The upack tool (upack.exe on Windows) can interact with a server’s machine-level Universal Package Registry in two ways.

The upack install command writes to the machine’s package registry, unless the --unregistered flag is specified; the user can specify a reason with the --comment option, or choose the user’s registry with the --user option.

The upack list command displays the currently registered package names and version numbers in the machine’s package registry. If the --user option is specified, the user registry is queried instead.

Implementation: Otter

Otter has three different operations that interact with a server’s machine-level Universal Package Registry. These describe the current behavior, and new behaviors may be added based on demand (such as utilizing package caching):

  • Push-Package – Ensures that the contents of a ProGet package are in the specified directory
  • Get-Package – this will download a universal package from a universal feed and install it to the specified directory; the package registry’s metadata will then be updated
  • Ensure-Package – this will compare the files on disk with the contents of a universal package in a universal feed and, if they are different, perform the steps that a Get-Package operation will do

Implementation: BuildMaster

BuildMaster has the same Push-Package, Ensure-Package, and Get-Package operations as Otter.

Example installedPackages.json Specification

[ { "name": "full/package/name", "version": "1.0.0", "path": "C:\\hdars" }, { "name": "hdars1000", "version": "1.0.1", "path": "C:\\hdars1000" } ]