[vcpkg] Add commands to maintain and verify versions db integrity (#14999)

* [vcpkg] Add x-ci-verify-versions command

* Code cleanup

* Remove port version splitting from x-history

* Fix wrong message on success

* Parallelize versions file generator

* Use cpu_count()/2 to avoid crashes

* Check db SHA against local SHA

* Check baseline version with x-ci-verify-versions and make baseline generator much faster

* Implement x-add-version to update version db files

* Better checks for x-add-info and make x-ci-verify-versions silent on success

* Use find() instead of [] on maps

* Create version file if does not exist

* Allow redirection of ports/ and port_versions/

* add test ports

* WIP end-to-end tests

* Change pats in e2e tests

* Fix e2e args

* e2e once more

* Pass ersions feature flag to e2e

* Exit with code 1 if there are errors

* Files to test for failure cases

* Update test files

* Add test for x-add-version

* fix redirected ports in last test

* Add CI check (use dummy data)

* Add feature-flags=versions

* Ignore subdirectories inside ports

* Add --verify-git-trees switch

* [vcpkg] Fix build breaks

* [x-ci-verify-versions] PR comments

* [x-add-version] PR comments

* Fix merge conflicts

* Modify tests and pipeline

* Baselines should only have version-string

* Refactor x-add-version

* [vcpkg] Fix help message

* [vcpkg] Fix minor warnings

* `x-add-version --all` doesn't stop on first failure and reduced default verbosity

* [vcpkg] Fix default-baseline

* Load file instead of using paths provider

* Format

* Remove ci test

* Add fish port for testing

* Update version files

* Update fish port to cause SHA discrepancy

* Test for discrepancy between local SHA and declared SHA

* Missing = operator

* Check for error message since x-add-version exits with code 0

* Make x-add-version fail with non-zero exit code if not run with --all

Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <ras0219@outlook.com>

4 files changed, 35 insertions, 0 deletions

diff --git a/toolsrc/include/vcpkg/commands.add-version.h b/toolsrc/include/vcpkg/commands.add-version.h
new file mode 100644
index 000000000..8b9e0336e
--- /dev/null
+++ b/toolsrc/include/vcpkg/commands.add-version.h
@@ -0,0 +1,13 @@
+#pragma once
+
+#include <vcpkg/commands.interface.h>
+
+namespace vcpkg::Commands::AddVersion
+{
+    void perform_and_exit(const VcpkgCmdArguments& args, const VcpkgPaths& paths);
+
+    struct AddVersionCommand : PathsCommand
+    {
+        void perform_and_exit(const VcpkgCmdArguments& args, const VcpkgPaths& paths) const override;
+    };
+}
\ No newline at end of file
diff --git a/toolsrc/include/vcpkg/commands.civerifyversions.h b/toolsrc/include/vcpkg/commands.civerifyversions.h
new file mode 100644
index 000000000..17d8f5613
--- /dev/null
+++ b/toolsrc/include/vcpkg/commands.civerifyversions.h
@@ -0,0 +1,13 @@
+#pragma once
+
+#include <vcpkg/commands.interface.h>
+
+namespace vcpkg::Commands::CIVerifyVersions
+{
+    void perform_and_exit(const VcpkgCmdArguments& args, const VcpkgPaths& paths);
+
+    struct CIVerifyVersionsCommand : PathsCommand
+    {
+        void perform_and_exit(const VcpkgCmdArguments& args, const VcpkgPaths& paths) const override;
+    };
+}
\ No newline at end of file
diff --git a/toolsrc/include/vcpkg/registries.h b/toolsrc/include/vcpkg/registries.h
index 006bc789d..c36c0179d 100644
--- a/toolsrc/include/vcpkg/registries.h
+++ b/toolsrc/include/vcpkg/registries.h
@@ -9,8 +9,10 @@
 #include <vcpkg/base/stringview.h>
 #include <vcpkg/base/view.h>
 
+#include <vcpkg/versiondeserializers.h>
 #include <vcpkg/versiont.h>
 
+#include <map>
 #include <memory>
 #include <string>
 #include <system_error>
@@ -103,4 +105,9 @@ namespace vcpkg
 
     std::unique_ptr<Json::IDeserializer<std::vector<Registry>>> get_registry_array_deserializer(
         const fs::path& configuration_directory);
+
+    ExpectedS<std::vector<std::pair<VersionT, std::string>>> get_builtin_versions(const VcpkgPaths& paths,
+                                                                                  StringView port_name);
+
+    ExpectedS<std::map<std::string, VersionT, std::less<>>> get_builtin_baseline(const VcpkgPaths& paths);
 }
diff --git a/toolsrc/include/vcpkg/vcpkgpaths.h b/toolsrc/include/vcpkg/vcpkgpaths.h
index 1f562c151..a175d51e2 100644
--- a/toolsrc/include/vcpkg/vcpkgpaths.h
+++ b/toolsrc/include/vcpkg/vcpkgpaths.h
@@ -122,6 +122,8 @@ namespace vcpkg
         fs::path git_checkout_port(Files::Filesystem& filesystem, StringView port_name, StringView git_tree) const;
         ExpectedS<std::string> git_show(const std::string& treeish, const fs::path& dot_git_dir) const;
 
+        ExpectedS<std::map<std::string, std::string, std::less<>>> git_get_local_port_treeish_map() const;
+
         Optional<const Json::Object&> get_manifest() const;
         Optional<const fs::path&> get_manifest_path() const;
         const Configuration& get_configuration() const;