[tensorflow-cc] More or less completely rewritten port, update to 2.3.1 (#13028)

* fix build issues occurring from default bazel config being used because customized bazel config is stored in wrong directory

* [tensorflow-cc] Update CONTROL and ci.baseline.txt

* fix also applies to windows static build

* fix debug and static builds as well as library naming for non-linux
- add patch to fix debug builds
- add patch to fix exports for static linking
- really build debug (instead of cloning release)
- override bazel build options for debug (work around bazel bug)
- bazel doesn't support static libraries: work around by building dynamic library and constructing static linkage commands from build log
- Windows .pdb file can't be >4GB even on x64: work around using reduced debug information
- Windows doesn't support .lib files >4GB even on x64, so split into multiple libs
- vcpkg requires equal amount of libs for debug and release: work around using handcrafted empty dummy libs
- fix naming of libs (.dll on Windows and .dylib on macOS)

* upgrade tensorflow from v1.14 to v2.3
- adapt patch files to tensorflow code changes
- update bazel from v0.25.2 to v3.1
- on Windows use python installed on the host instead of embedded python obtained via vcpkg because embedded python lacks pip, which we need to obtain numpy
- on Windows add MSYS2 to the PATH so that bazel tools can access MSYS2 GIT
- add support for custom CA certificates when using HTTPS_PROXY

* fix execute process macro
The existing implementation totally screwed up commands if the command's arguments contained semicolons (this is the case, e.g., in the FindPython modules of the cmake distribution).

* extend overriden execute_process to more than one COMMAND as there actually are use cases for this

* added another patch required for tensorflow v2.3, fixed path and working directory

* Revert "incorporate changes from microsoft:master"

* Revert "Revert "incorporate changes from microsoft:master""

* final fixes for static build + improving out messages

* enabling linux and osx in CI to see if it works now

* simplified code, fixed version numbers, fixed generated include cmake file

* fix failing postbuild check on handcrafted empty dummy library by spreading the last real libraries contents over the required number of libraries

* remove dead code commit by mistake again

* improvements from code review

* cleaner fix for debug code

* find pip3 in PATH (PYTHON3_DIR apparently not valid for pip3)

* fix error in python helper script

* fix wrong libname in postbuild script

* fix python detection + switch to python on msys2 (instead of embedded python) for Windows as we need numpy

* fix order of arguments

* fix command (it may contain spaces such as C:\Program Files\...)

* revert last commit (root cause for CI failures is something different: there are line breaks in path)

* fix regex comparision
(value needs to be escaped as it may contains regex special characters such as brackets, eg C:/Program Files (x86)/...)

* fix linebreaks in generated file

* fix CRT linkage
(macOS doesn't support static CRT linkage; it's set to dynamic even static target triplets for macOS and linux)

* refactor implemenation to avoid as much code duplication as possible -- algorithmically identical

* fix version numbers in helper scripts

* enable work-around for Windows until bazel fix is available

* install missing python3-pip on linux

* fix linux build by patching

* apply timeout feature now available via merged master branch

* correct linux build patch

* improve debug build patches
(no functional difference because LOG(FATAL, ...) macro internally anyway calls abort(), which the compiler doesn't detect in debug mode...

* improve linux patch

* temporarily add debug to inspect what's going on on macOS CI

* remove temporary debug code and fix static linking scripts for linux and macOS

* fix regex escaping

* fix ambiguous match while grepping for the framework link command

* extend fix of ambiguous match while grepping for the framework link command

* fix what merge of master broke

* fix more what got broken by merging master
(all packages and their dependencies are now maintained manually instead of using pacman...)

* remove "unofficial" from filename

* added switch do distinct classic and manifest mode when generating config.cmake file

* create symlinks for libraries without version number

* fix linux postbuild script

* temporarily disable code making problems

* add note for linking on Linux and macOS

* forget to add README file in previous commit

* add file forgotton in macro fixing patch

* fix python library path

* fix macOS static link command

* update linkage instructions in README

* Update ports/tensorflow-cc/CONTROL

* Update ports/tensorflow-cc/portfile.cmake

* Update scripts/ci.baseline.txt

* use vcpkg_execute_required_process

* pass C_FLAGS and CXX_FLAGS to bazel

* fix INTERFACE_INCLUDE_DIRECTORIES

* fix optional c/cxx arguments

* also add linker opts

* update README

* merge static libs into one
to support force_load (cannot force_load both due to duplicate symbols)

* update README

* quote python path (it might contain spaces that don't get escaped inside outer quotes of bash command)

* fix python path also for static build

* add arm(64) as currently unsupported arch

* bazel 3.7 is available -> remove workaround

* update README, remove necessary c-ares from deps

* update msys package

* add uwp specific options, and minor general improvements

* fix string replace

* fix control file and windows path separator

* revert backslashes-fix -- the root cause was missing .exe extension

* upgrade to tf 2.3.1

* fix hard-coded version

* remove uwp work-in-progress code so that PR can be merged

* update README and print out usage info in portfile

* Update ports/tensorflow-cc/README-linux

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* Update ports/tensorflow-cc/README-linux

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* Update ports/tensorflow-cc/README-linux

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* Update ports/tensorflow-cc/README-linux

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* Update ports/tensorflow-cc/README-linux

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* Update ports/tensorflow-cc/README-windows

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* Update ports/tensorflow-cc/portfile.cmake

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* Update ports/tensorflow-cc/portfile.cmake

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* Update ports/tensorflow-cc/portfile.cmake

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* Update ports/tensorflow-cc/portfile.cmake

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* Update ports/tensorflow-cc/portfile.cmake

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* improve usage hints as discussed in review

* add comment

* apply changes from review

* make additional compiler / linker args space-proof

* Update ports/tensorflow-cc/README-macos

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* Update ports/tensorflow-cc/tensorflow-cc-config-shared.cmake.in

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* Update ports/tensorflow-cc/tensorflow-cc-config-shared.cmake.in

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* Update ports/tensorflow-cc/README-linux

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* Update ports/tensorflow-cc/README-macos

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* apply changes from code review

* maybe fix the config files

* rob.maynard CRs

* fix windows static lib naming for first part

* Update ports/tensorflow-cc/generate_static_link_cmd_windows.py

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

* apply rob maynards 1st suggestion

* nicole CRs

* format/fix-compile

* fix missing string termination

* prefer IMPORTED_LOCATION over IMPORTED_LOCATION_RELEASE to have default fall-back

* hopefully fix the issue where no libraries are generated

* final stuff

Co-authored-by: Gehweiler <Joachim_Gehweiler@McAfee.com>
Co-authored-by: wangli28 <wangli28@beyondsoft.com>
Co-authored-by: Jack·Boos·Yu <47264268+JackBoosY@users.noreply.github.com>
Co-authored-by: Joachim Gehweiler <joachim@Joachims-iMac.local>
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

3 files changed, 20 insertions, 17 deletions

diff --git a/toolsrc/src/vcpkg/base/files.cpp b/toolsrc/src/vcpkg/base/files.cpp
index 4cccebd45..891207445 100644
--- a/toolsrc/src/vcpkg/base/files.cpp
+++ b/toolsrc/src/vcpkg/base/files.cpp
@@ -26,20 +26,6 @@
 
 namespace
 {
-#if defined(_WIN32)
-    struct IsSlash
-    {
-        bool operator()(const wchar_t c) const noexcept { return c == L'/' || c == L'\\'; }
-    };
-#else
-    struct IsSlash
-    {
-        bool operator()(const char c) const noexcept { return c == '/'; }
-    };
-#endif
-
-    constexpr IsSlash is_slash;
-
     struct NativeStringView
     {
         const fs::path::value_type* first;
diff --git a/toolsrc/src/vcpkg/build.cpp b/toolsrc/src/vcpkg/build.cpp
index c20f48b36..bae937277 100644
--- a/toolsrc/src/vcpkg/build.cpp
+++ b/toolsrc/src/vcpkg/build.cpp
@@ -203,6 +203,7 @@ namespace vcpkg::Build
     static const std::string NAME_EMPTY_PACKAGE = "PolicyEmptyPackage";
     static const std::string NAME_DLLS_WITHOUT_LIBS = "PolicyDLLsWithoutLIBs";
     static const std::string NAME_DLLS_WITHOUT_EXPORTS = "PolicyDLLsWithoutExports";
+    static const std::string NAME_MISMATCHED_NUMBER_OF_BINARIES = "PolicyMismatchedNumberOfBinaries";
     static const std::string NAME_ONLY_RELEASE_CRT = "PolicyOnlyReleaseCRT";
     static const std::string NAME_EMPTY_INCLUDE_FOLDER = "PolicyEmptyIncludeFolder";
     static const std::string NAME_ALLOW_OBSOLETE_MSVCRT = "PolicyAllowObsoleteMsvcrt";
@@ -210,6 +211,19 @@ namespace vcpkg::Build
     static const std::string NAME_SKIP_DUMPBIN_CHECKS = "PolicySkipDumpbinChecks";
     static const std::string NAME_SKIP_ARCHITECTURE_CHECK = "PolicySkipArchitectureCheck";
 
+    static std::remove_const_t<decltype(ALL_POLICIES)> generate_all_policies()
+    {
+        std::remove_const_t<decltype(ALL_POLICIES)> res;
+        for (size_t i = 0; i < res.size(); ++i)
+        {
+            res[i] = static_cast<BuildPolicy>(i);
+        }
+
+        return res;
+    }
+
+    decltype(ALL_POLICIES) ALL_POLICIES = generate_all_policies();
+
     const std::string& to_string(BuildPolicy policy)
     {
         switch (policy)
@@ -217,6 +231,7 @@ namespace vcpkg::Build
             case BuildPolicy::EMPTY_PACKAGE: return NAME_EMPTY_PACKAGE;
             case BuildPolicy::DLLS_WITHOUT_LIBS: return NAME_DLLS_WITHOUT_LIBS;
             case BuildPolicy::DLLS_WITHOUT_EXPORTS: return NAME_DLLS_WITHOUT_EXPORTS;
+            case BuildPolicy::MISMATCHED_NUMBER_OF_BINARIES: return NAME_MISMATCHED_NUMBER_OF_BINARIES;
             case BuildPolicy::ONLY_RELEASE_CRT: return NAME_ONLY_RELEASE_CRT;
             case BuildPolicy::EMPTY_INCLUDE_FOLDER: return NAME_EMPTY_INCLUDE_FOLDER;
             case BuildPolicy::ALLOW_OBSOLETE_MSVCRT: return NAME_ALLOW_OBSOLETE_MSVCRT;
@@ -234,6 +249,7 @@ namespace vcpkg::Build
             case BuildPolicy::EMPTY_PACKAGE: return "VCPKG_POLICY_EMPTY_PACKAGE";
             case BuildPolicy::DLLS_WITHOUT_LIBS: return "VCPKG_POLICY_DLLS_WITHOUT_LIBS";
             case BuildPolicy::DLLS_WITHOUT_EXPORTS: return "VCPKG_POLICY_DLLS_WITHOUT_EXPORTS";
+            case BuildPolicy::MISMATCHED_NUMBER_OF_BINARIES: return "VCPKG_POLICY_MISMATCHED_NUMBER_OF_BINARIES";
             case BuildPolicy::ONLY_RELEASE_CRT: return "VCPKG_POLICY_ONLY_RELEASE_CRT";
             case BuildPolicy::EMPTY_INCLUDE_FOLDER: return "VCPKG_POLICY_EMPTY_INCLUDE_FOLDER";
             case BuildPolicy::ALLOW_OBSOLETE_MSVCRT: return "VCPKG_POLICY_ALLOW_OBSOLETE_MSVCRT";
@@ -1221,7 +1237,7 @@ namespace vcpkg::Build
         if (!version.empty()) build_info.version = std::move(version);
 
         std::map<BuildPolicy, bool> policies;
-        for (auto policy : G_ALL_POLICIES)
+        for (auto policy : ALL_POLICIES)
         {
             const auto setting = parser.optional_field(to_string(policy));
             if (setting.empty()) continue;
diff --git a/toolsrc/src/vcpkg/postbuildlint.cpp b/toolsrc/src/vcpkg/postbuildlint.cpp
index 7b8e65ed6..8490ac364 100644
--- a/toolsrc/src/vcpkg/postbuildlint.cpp
+++ b/toolsrc/src/vcpkg/postbuildlint.cpp
@@ -872,7 +872,7 @@ namespace vcpkg::PostBuildLint
         std::vector<fs::path> release_libs = fs.get_files_recursive(release_lib_dir);
         Util::erase_remove_if(release_libs, not_extension_pred(fs, ".lib"));
 
-        if (!pre_build_info.build_type)
+        if (!pre_build_info.build_type && !build_info.policies.is_enabled(BuildPolicy::MISMATCHED_NUMBER_OF_BINARIES))
             error_count += check_matching_debug_and_release_binaries(debug_libs, release_libs);
 
         if (!build_info.policies.is_enabled(BuildPolicy::SKIP_ARCHITECTURE_CHECK))
@@ -892,7 +892,8 @@ namespace vcpkg::PostBuildLint
         {
             case Build::LinkageType::DYNAMIC:
             {
-                if (!pre_build_info.build_type)
+                if (!pre_build_info.build_type &&
+                    !build_info.policies.is_enabled(BuildPolicy::MISMATCHED_NUMBER_OF_BINARIES))
                     error_count += check_matching_debug_and_release_binaries(debug_dlls, release_dlls);
 
                 error_count += check_lib_files_are_available_if_dlls_are_available(