[docs registries] Add reference documentation (#17672)

* using registries reference documentation

* start work on "creating registries" document

also fix minor issue in "using registries" doc

* Robert's comments, continue working

* finish creating registries docs

* add links to example registries

* aupopa cr

3 files changed, 494 insertions, 0 deletions

diff --git a/docs/README.md b/docs/README.md
index a0a53b1cc..95b26f903 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -23,6 +23,7 @@ Vcpkg helps you manage C and C++ libraries on Windows, Linux and MacOS. This too
 - [Usage with Android](users/android.md)
 - [Usage with Mingw-w64](users/mingw.md)
 - [Host Dependencies](users/host-dependencies.md)
+- [Using Registries](users/registries.md)
 
 ### Maintainer Help
 
@@ -31,6 +32,7 @@ Vcpkg helps you manage C and C++ libraries on Windows, Linux and MacOS. This too
 - [Portfile functions](maintainers/portfile-functions.md)
 - [Common CMake definitions](maintainers/vcpkg_common_definitions.md)
 - [Maintainer Guidelines](maintainers/maintainer-guide.md)
+- [Creating Registries](maintainers/registries.md)
 
 ### [Vcpkg-Tool](https://github.com/microsoft/vcpkg-tool) Maintainer Help
 
diff --git a/docs/maintainers/registries.md b/docs/maintainers/registries.md
new file mode 100644
index 000000000..bef296ea3
--- /dev/null
+++ b/docs/maintainers/registries.md
@@ -0,0 +1,357 @@
+# Creating Registries
+
+**The latest version of this documentation is available on [GitHub](https://github.com/Microsoft/vcpkg/tree/master/docs/maintainers/registries.md).**
+
+There are two parts to using registries; this documents the creation side of
+the relationship. In order to learn more about using registries that others
+have created, please read [this documentation](../users/registries.md).
+## Table of Contents
+
+- [Creating Registries](#creating-registries)
+  - [Table of Contents](#table-of-contents)
+  - [Overview](#overview)
+    - [Git Registries](#git-registries)
+      - [Adding a New Version](#adding-a-new-version)
+    - [Filesystem Registries](#filesystem-registries)
+      - [Adding a New Version](#adding-a-new-version-1)
+
+## Overview
+
+Registries are collections of ports and their versions. There are two major
+choices of implementation for registries, if you want to create your own -
+git registries, and filesystem registries.
+
+Git registries are simple git repositories, and can be shared publicly or
+privately via normal mechanisms for git repositories. The vcpkg repository at
+<https://github.com/microsoft/vcpkg>, for example, is a git registry.
+
+Filesystem registries are designed as more of a testing ground. Given that they
+literally live on your filesystem, the only way to share them is via shared
+directories. However, filesystem registries can be useful as a way to represent
+registries held in non-git version control systems, assuming one has some way
+to get the registry onto the disk.
+
+Note that we expect the set of registry types to grow over time; if you would
+like support for registries built in your favorite public version control
+system, don't hesitate to open a PR.
+
+The basic structure of a registry is:
+
+- The set of versions that are considered "latest" at certain times in history,
+  known as the "baseline".
+- The set of all the versions of all the ports, and where to find each of
+  these in the registry.
+
+### Git Registries
+
+As you're following along with this documentation, it may be helpful to have
+a working example to refer to. We've written one and put it here:
+<https://github.com/northwindtraders/vcpkg-registry>.
+
+All git registries must have a `versions/baseline.json` file. This file
+contains the set of "latest versions" at a certain commit. It is laid out as
+a top-level object containing only the `"default"` field. This field should
+contain an object mapping port names to the version which is currently the
+latest.
+
+Here's an example of a valid baseline.json:
+
+```json
+{
+  "default": {
+    "kitten": {
+      "baseline": "2.6.2",
+      "port-version": 0
+    },
+    "port-b": {
+      "baseline": "19.00",
+      "port-version": 2
+    }
+  }
+}
+```
+
+The `versions` directory contains all the information about which versions of
+which packages are contained in the registry, along with where those versions
+are stored. The rest of the registry just acts as a backing store, as far as
+vcpkg is concerned: only things inside the `versions` directory will be used
+to direct how your registry is seen by vcpkg.
+
+Each port in a registry should exist in the versions directory as
+`<first letter of port>-/<name of port>.json`; in other words, the
+information about the `kitten` port would be located in
+`versions/k-/kitten.json`. This should be a top-level object with only a
+single field: `"versions"`. This field should contain an array of version 
+objects:
+
+- The version of the port in question; should be exactly the same as the
+  `vcpkg.json` file, including the version fields and `"port-version"`.
+- The `"git-tree"` field, which is a git tree; in other words, what you get
+  when you write `git rev-parse COMMIT-ID:path/to/port`.
+
+Note that the version fields for ports with `CONTROL` files, is 
+`"version-string"`; we do not recommend using `CONTROL` files in new
+registries, however.
+
+_WARNING_: One very important part of registries is that versions should
+_never_ be changed. Updating to a later ref should never remove or change an
+existing version. It must always be safe to update a registry.
+
+Here's an example of a valid version database for a `kitten` port with one 
+version:
+
+```json
+{
+  "versions": [
+    {
+      "version": "2.6.2",
+      "port-version": 0,
+      "git-tree": "67d60699c271b7716279fdea5a5c6543929eb90e"
+    }
+  ]
+}
+```
+
+In general, it's not important where you place port directories. However, the
+idiom in vcpkg is to follow what the built in vcpkg registry does: your 
+`kitten` port should be placed in `ports/kitten`.
+
+_WARNING_: One other thing to keep in mind is that when you update a registry,
+all previous versions should also be accessible. Since your user will set their
+baseline to a commit ID, that commit ID must always exist, and be accessible
+from your HEAD commit, which is what is actually fetched. This means that your
+HEAD commit should be a child of all previous HEAD commits.
+
+#### Adding a New Version
+
+There is some git trickery involved in creating a new version of a port. The
+first thing to do is make some changes, update the `"port-version"` and regular
+version field as you need to, and then test with `overlay-ports`:
+`vcpkg install kitten --overlay-ports=ports/kitten`.
+
+Once you've finished your testing, you'll need to make sure that the directory
+as it is is under git's purview. You'll do this by creating a temporary commit:
+
+```pwsh
+> git add ports/kitten
+> git commit -m 'temporary commit'
+```
+
+Then, get the git tree ID of the directory:
+
+```pwsh
+> git rev-parse HEAD:ports/kitten
+73ad3c823ef701c37421b450a34271d6beaf7b07
+```
+
+Then, you can add this version to the versions database. At the top of your
+`versions/k-/kitten.json`, you can add (assuming you're adding version
+`2.6.3#0`):
+
+```json
+{
+  "versions": [
+    {
+      "version": "2.6.3",
+      "port-version": 0,
+      "git-tree": "73ad3c823ef701c37421b450a34271d6beaf7b07"
+    },
+    {
+      "version": "2.6.2",
+      "port-version": 0,
+      "git-tree": "67d60699c271b7716279fdea5a5c6543929eb90e"
+    }
+  ]
+}
+```
+
+then, you'll want to modify your `versions/baseline.json` with your new version 
+as well:
+
+```json
+{
+  "default": {
+    "kitten": {
+      "baseline": "2.6.3",
+      "port-version": 0
+    },
+    "port-b": {
+      "baseline": "19.00",
+      "port-version": 2
+    }
+  }
+}
+```
+
+and amend your current commit:
+
+```pwsh
+> git commit --amend
+```
+
+then share away!
+
+### Filesystem Registries
+
+As you're following along with this documentation, it may be helpful to have
+a working example to refer to. We've written one and put it here:
+<https://github.com/vcpkg/example-filesystem-registry>.
+
+All filesystem registries must have a `versions/baseline.json` file. This file
+contains the set of "latest versions" for a certain version of the registry.
+It is laid out as a top-level object containing a map from version name to
+"baseline objects", which map port names to the version which is considered
+"latest" for that version of the registry.
+
+Filesystem registries need to decide on a versioning scheme. Unlike git 
+registries, which have the implicit versioning scheme of refs, filesystem
+registries can't rely on the version control system here. One possible option
+is to do a daily release, and have your "versions" be dates.
+
+_WARNING_: A baseline must always refer to the same set of versions. If you
+want to add new versions, you need to create a new version of the registry in
+the `baseline.json` file.
+
+Here's an example of a valid `baseline.json`, for a registry that has decided
+upon dates for their versions:
+
+```json
+{
+  "2021-04-16": {
+    "kitten": {
+      "baseline": "2.6.2",
+      "port-version": 0
+    },
+    "port-b": {
+      "baseline": "19.00",
+      "port-version": 2
+    }
+  },
+  "2021-04-15": {
+    "kitten": {
+      "baseline": "2.6.2",
+      "port-version": 0
+    },
+    "port-b": {
+      "baseline": "19.00",
+      "port-version": 1
+    }
+  }
+}
+```
+
+The `versions` directory contains all the information about which versions of
+which packages are contained in the registry, along with where those versions
+are stored. The rest of the registry just acts as a backing store, as far as
+vcpkg is concerned: only things inside the `versions` directory will be used
+to direct how your registry is seen by vcpkg.
+
+Each port in a registry should exist in the versions directory as
+`<first letter of port>-/<name of port>.json`; in other words, the
+information about the `kitten` port would be located in
+`versions/k-/kitten.json`. This should be a top-level object with only a
+single field: `"versions"`. This field should contain an array of version 
+objects:
+
+- The version of the port in question; should be exactly the same as the
+  `vcpkg.json` file, including the version fields and `"port-version"`.
+- The `"path"` field: a relative directory, rooted at the base of the registry
+  (in other words, the directory where `versions` is located), to the port 
+  directory. It should look something like `"$/path/to/port/dir`"
+
+Note that the version fields for ports with `CONTROL` files, is 
+`"version-string"`; we do not recommend using `CONTROL` files in new
+registries, however.
+
+In general, it's not important where you place port directories. However, the
+idiom in vcpkg is to follow somewhat closely to what the built in vcpkg
+registry does: your `kitten` port at version `x.y.z` should be placed in
+`ports/kitten/x.y.z`, with port versions appended as you see fit (although
+since `#` is not a good character to use for file names, perhaps use `_`).
+
+_WARNING_: One very important part of registries is that versions should
+_never_ be changed. One should never remove or change an existing version.
+Your changes to your registry shouldn't change behavior to downstream users.
+
+Here's an example of a valid version database for a `kitten` port with one 
+version:
+
+```json
+{
+  "versions": [
+    {
+      "version": "2.6.2",
+      "port-version": 0,
+      "git-tree": "$/ports/kitten/2.6.2_0"
+    }
+  ]
+}
+```
+
+#### Adding a New Version
+
+Unlike git registries, adding a new version to a filesystem registry mostly
+involves a lot of copying. The first thing to do is to copy the latest
+version of your port into a new version directory, update the version and
+`"port-version"` fields as you need to, and then test with `overlay-ports`:
+`vcpkg install kitten --overlay-ports=ports/kitten/new-version`.
+
+Once you've finished your testing, you can add this new version to the top of
+your `versions/k-/kitten.json`:
+
+```json
+{
+  "versions": [
+    {
+      "version": "2.6.3",
+      "port-version": 0,
+      "git-tree": "$/ports/kitten/2.6.3_0"
+    },
+    {
+      "version": "2.6.2",
+      "port-version": 0,
+      "git-tree": "$/ports/kitten/2.6.2_0"
+    }
+  ]
+}
+```
+
+then, you'll want to modify your `versions/baseline.json` with your new version 
+as well (remember not to modify existing baselines):
+
+```json
+{
+  "2021-04-17": {
+    "kitten": {
+      "baseline": "2.6.3",
+      "port-version": 0
+    },
+    "port-b": {
+      "baseline": "19.00",
+      "port-version": 2
+    }
+  },
+  "2021-04-16": {
+    "kitten": {
+      "baseline": "2.6.2",
+      "port-version": 0
+    },
+    "port-b": {
+      "baseline": "19.00",
+      "port-version": 2
+    }
+  },
+  "2021-04-15": {
+    "kitten": {
+      "baseline": "2.6.2",
+      "port-version": 0
+    },
+    "port-b": {
+      "baseline": "19.00",
+      "port-version": 1
+    }
+  }
+}
+```
+
+and you're done!
\ No newline at end of file
diff --git a/docs/users/registries.md b/docs/users/registries.md
new file mode 100644
index 000000000..53db7cb1f
--- /dev/null
+++ b/docs/users/registries.md
@@ -0,0 +1,135 @@
+# Using Registries
+
+**The latest version of this documentation is available on [GitHub](https://github.com/Microsoft/vcpkg/tree/master/docs/users/registries.md).**
+
+There are two parts to using registries; this documents the use side of the
+relationship. In order to learn more about creating registries for others to
+use, please read [this documentation](../maintainers/registries.md).
+
+## Table of Contents
+
+- [Using Registries](#using-registries)
+  - [Table of Contents](#table-of-contents)
+  - [`vcpkg-configuration.json`](#vcpkg-configurationjson)
+    - [Registry Objects](#registry-objects)
+      - [Registry Objects: `"kind"`](#registry-objects-kind)
+      - [Registry Objects: `"baseline"`](#registry-objects-baseline)
+      - [Registry Objects: `"repository"`](#registry-objects-repository)
+      - [Registry Objects: `"path"`](#registry-objects-path)
+    - [Configuration: `"default-registry"`](#configuration-default-registry)
+    - [Configuration: `"registries"`](#configuration-registries)
+    - [Example Configuration File](#example-configuration-file)
+  - [Package Name Resolution](#package-name-resolution)
+    - [Versioning Support](#versioning-support)
+
+## `vcpkg-configuration.json`
+
+From a high level perspective, everything that a project needs to define
+about registries is contained in the vcpkg configuration file. In classic
+mode, the configuration file lies in the vcpkg root; for manifest mode,
+the file must exist next to the project's `vcpkg.json` file.
+This file is named `vcpkg-configuration.json`, and it's a simple top-level
+object file.
+
+### Registry Objects
+
+Registries are defined in JSON as objects. They must contain at least the
+`"kind"` and `"baseline"` fields, and additionally the different kinds of
+registry will have their own way of defining where the registry can be found:
+
+- git registries require the `"repository"` field
+- filesystem registries require the `"path"` field
+- built-in registries do not require a field, since there is only one
+  built-in registry.
+
+#### Registry Objects: `"kind"`
+
+The `"kind"` field must be a string:
+
+- For git registries: `"git"`
+- For filesystem registries: `"filesystem"`
+- For the builtin registry: `"builtin"`
+
+#### Registry Objects: `"baseline"`
+
+The `"baseline"` field must be a string. For git registries and for the 
+built-in registry, it should be a 40-character commit ID.
+For filesystem registries, it can be any string that the registry defines.
+
+#### Registry Objects: `"repository"`
+
+This should be a string, of any repository format that git understands:
+
+- `"https://github.com/microsoft/vcpkg"`
+- `"git@github.com:microsoft/vcpkg"`
+- `"/dev/vcpkg-registry"`
+
+#### Registry Objects: `"path"`
+
+This should be a path; it can be either absolute or relative; relative paths
+will be based at the directory the `vcpkg-configuration.json` lives in.
+
+### Configuration: `"default-registry"`
+
+The `"default-registry"` field should be a registry object. It defines
+the registry that is used for all packages that are not claimed by any
+package registries. It may also be `null`, in which case no packages that
+are not claimed by package registries may be installed.
+
+### Configuration: `"registries"`
+
+The `"registries"` field should be an array of registry objects, each of
+which additionally contain a `"packages"` field, which should be an array of
+package names. These define the package registries, which are used for 
+the specific packages named by the `"packages"` field.
+
+The `"packages"` fields of all the package registries must be disjoint.
+
+### Example Configuration File
+
+Let's assume that you have mirrored <https://github.com/microsoft/vcpkg> at
+<https://git.example.com/vcpkg>: this will be your default registry.
+Additionally, you want to use North Wind Trader's registry for their
+beison and beicode libraries. The following `vcpkg-configuration.json`
+will work:
+
+```json
+{
+  "default-registry": {
+    "kind": "git",
+    "repository": "https://git.example.com/vcpkg",
+    "baseline": "eefee7408133f3a0fef711ef9c6a3677b7e06fd7"
+  },
+  "registries": [
+    {
+      "kind": "git",
+      "repository": "https://github.com/northwindtraders/vcpkg-registry",
+      "baseline": "dacf4de488094a384ca2c202b923ccc097956e0c",
+      "packages": [ "beicode", "beison" ]
+    }
+  ]
+}
+```
+
+## Package Name Resolution
+
+The way package name resolution works in vcpkg is fairly distinct from many
+package managers. It is very carefully designed to _never_ implicitly choose
+the registry that a package is fetched from. Just from
+`vcpkg-configuration.json`, one can tell exactly from which registry a
+package definition will be fetched from.
+
+The name resolution algorithm is as follows:
+
+- If there is a package registry that claims the package name,
+  use that registry; otherwise
+- If there is a default registry defined, use that registry; otherwise
+- If the default registry is set to `null`, error out; otherwise
+- use the built-in registry.
+
+### Versioning Support
+
+Versioning with custom registries works exactly as it does in the built-in
+registry. You can read more about that in the [versioning documentation].
+
+[versioning documentation]: versioning.md
\ No newline at end of file