[tensorflow] Add C port (#14392)

* fix build issues occurring from default bazel config being used because customized bazel config is stored in wrong directory

* [tensorflow-cc] Update CONTROL and ci.baseline.txt

* fix also applies to windows static build

* fix debug and static builds as well as library naming for non-linux
- add patch to fix debug builds
- add patch to fix exports for static linking
- really build debug (instead of cloning release)
- override bazel build options for debug (work around bazel bug)
- bazel doesn't support static libraries: work around by building dynamic library and constructing static linkage commands from build log
- Windows .pdb file can't be >4GB even on x64: work around using reduced debug information
- Windows doesn't support .lib files >4GB even on x64, so split into multiple libs
- vcpkg requires equal amount of libs for debug and release: work around using handcrafted empty dummy libs
- fix naming of libs (.dll on Windows and .dylib on macOS)

* upgrade tensorflow from v1.14 to v2.3
- adapt patch files to tensorflow code changes
- update bazel from v0.25.2 to v3.1
- on Windows use python installed on the host instead of embedded python obtained via vcpkg because embedded python lacks pip, which we need to obtain numpy
- on Windows add MSYS2 to the PATH so that bazel tools can access MSYS2 GIT
- add support for custom CA certificates when using HTTPS_PROXY

* fix execute process macro
The existing implementation totally screwed up commands if the command's arguments contained semicolons (this is the case, e.g., in the FindPython modules of the cmake distribution).

* extend overriden execute_process to more than one COMMAND as there actually are use cases for this

* added another patch required for tensorflow v2.3, fixed path and working directory

* Revert "incorporate changes from microsoft:master"

* Revert "Revert "incorporate changes from microsoft:master""

* final fixes for static build + improving out messages

* enabling linux and osx in CI to see if it works now

* simplified code, fixed version numbers, fixed generated include cmake file

* fix failing postbuild check on handcrafted empty dummy library by spreading the last real libraries contents over the required number of libraries

* remove dead code commit by mistake again

* improvements from code review

* cleaner fix for debug code

* find pip3 in PATH (PYTHON3_DIR apparently not valid for pip3)

* fix error in python helper script

* fix wrong libname in postbuild script

* fix python detection + switch to python on msys2 (instead of embedded python) for Windows as we need numpy

* fix order of arguments

* fix command (it may contain spaces such as C:\Program Files\...)

* revert last commit (root cause for CI failures is something different: there are line breaks in path)

* fix regex comparision
(value needs to be escaped as it may contains regex special characters such as brackets, eg C:/Program Files (x86)/...)

* fix linebreaks in generated file

* fix CRT linkage
(macOS doesn't support static CRT linkage; it's set to dynamic even static target triplets for macOS and linux)

* refactor implemenation to avoid as much code duplication as possible -- algorithmically identical

* fix version numbers in helper scripts

* enable work-around for Windows until bazel fix is available

* install missing python3-pip on linux

* fix linux build by patching

* apply timeout feature now available via merged master branch

* correct linux build patch

* improve debug build patches
(no functional difference because LOG(FATAL, ...) macro internally anyway calls abort(), which the compiler doesn't detect in debug mode...

* improve linux patch

* temporarily add debug to inspect what's going on on macOS CI

* remove temporary debug code and fix static linking scripts for linux and macOS

* fix regex escaping

* fix ambiguous match while grepping for the framework link command

* extend fix of ambiguous match while grepping for the framework link command

* fix what merge of master broke

* fix more what got broken by merging master
(all packages and their dependencies are now maintained manually instead of using pacman...)

* remove "unofficial" from filename

* added switch do distinct classic and manifest mode when generating config.cmake file

* create symlinks for libraries without version number

* fix linux postbuild script

* temporarily disable code making problems

* add note for linking on Linux and macOS

* forget to add README file in previous commit

* add file forgotton in macro fixing patch

* fix python library path

* fix macOS static link command

* update linkage instructions in README

* Update ports/tensorflow-cc/CONTROL

* Update ports/tensorflow-cc/portfile.cmake

* Update scripts/ci.baseline.txt

* use vcpkg_execute_required_process

* pass C_FLAGS and CXX_FLAGS to bazel

* fix INTERFACE_INCLUDE_DIRECTORIES

* fix optional c/cxx arguments

* also add linker opts

* update README

* merge static libs into one
to support force_load (cannot force_load both due to duplicate symbols)

* update README

* quote python path (it might contain spaces that don't get escaped inside outer quotes of bash command)

* fix python path also for static build

* add arm(64) as currently unsupported arch

* bazel 3.7 is available -> remove workaround

* update README, remove necessary c-ares from deps

* update msys package

* add uwp specific options, and minor general improvements

* fix string replace

* fix control file and windows path separator

* revert backslashes-fix -- the root cause was missing .exe extension

* upgrade to tf 2.3.1

* fix hard-coded version

* remove uwp work-in-progress code so that PR can be merged

* add [tensorflow] C API port

* missing in previous commit

* fix include file

* remove unnecessary suffix

* update README and print out usage info in portfile

* Update ports/tensorflow-cc/CONTROL

Co-authored-by: NancyLi1013 <46708020+NancyLi1013@users.noreply.github.com>

* Update ports/tensorflow/portfile.cmake

Co-authored-by: NancyLi1013 <46708020+NancyLi1013@users.noreply.github.com>

* follow-up to code review

* fix suffix parameter

* fix quoting

* extend linux patch

* another try to fix quoting of possibly empty string parameter

* different approach to fix empty string arguments

* update list of headers for tensorflow 2.3.1 (was still 2.3.0)

* Update ports/tensorflow/CONTROL

Co-authored-by: NancyLi1013 <46708020+NancyLi1013@users.noreply.github.com>

* move common stuff into tensorflow-common as discussed in review

* Apply suggestions from code review

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

Co-authored-by: Gehweiler <Joachim_Gehweiler@McAfee.com>
Co-authored-by: wangli28 <wangli28@beyondsoft.com>
Co-authored-by: Jack·Boos·Yu <47264268+JackBoosY@users.noreply.github.com>
Co-authored-by: Joachim Gehweiler <joachim@Joachims-iMac.local>
Co-authored-by: NancyLi1013 <46708020+NancyLi1013@users.noreply.github.com>
Co-authored-by: nicole mazzuca <mazzucan@outlook.com>

1 files changed, 111 insertions, 0 deletions

diff --git a/ports/tensorflow-common/fix-dbg-build-errors.patch b/ports/tensorflow-common/fix-dbg-build-errors.patch
new file mode 100644
index 000000000..63e95dbb8
--- /dev/null
+++ b/ports/tensorflow-common/fix-dbg-build-errors.patch
@@ -0,0 +1,111 @@
+diff --git a/tensorflow/core/platform/path.cc b/tensorflow/core/platform/path.cc
+--- a/tensorflow/core/platform/path.cc
++++ b/tensorflow/core/platform/path.cc
+@@ -329,6 +329,7 @@
+   }
+   LOG(FATAL) << "No temp directory found.";
+ #endif
++  std::abort();
+ }
+ 
+ bool GetTestUndeclaredOutputsDir(string* dir) {
+
+diff --git a/tensorflow/core/framework/device_base.cc b/tensorflow/core/framework/device_base.cc
+--- a/tensorflow/core/framework/device_base.cc
++++ b/tensorflow/core/framework/device_base.cc
+@@ -65,10 +65,12 @@
+ 
+ const DeviceAttributes& DeviceBase::attributes() const {
+   LOG(FATAL) << "Device does not implement attributes()";
++  std::abort();
+ }
+ 
+ const string& DeviceBase::name() const {
+   LOG(FATAL) << "Device does not implement name()";
++  std::abort();
+ }
+ 
+ void DeviceBase::set_eigen_cpu_device(Eigen::ThreadPoolDevice* d) {
+
+diff --git a/tensorflow/core/kernels/depthtospace_op.cc b/tensorflow/core/kernels/depthtospace_op.cc
+--- a/tensorflow/core/kernels/depthtospace_op.cc
++++ b/tensorflow/core/kernels/depthtospace_op.cc
+@@ -117,13 +117,13 @@
+         // NCHW_VECT_C with 4 x qint8 can be treated as NCHW int32.
+         auto Tinput_v = input.template reinterpret_last_dimension<int32, 4>();
+         auto Toutput_v = outputs_tensor->reinterpret_last_dimension<int32, 4>();
+-        functor::DepthToSpaceOpFunctor<GPUDevice, int32, FORMAT_NCHW> functor;
+-        functor(context->eigen_device<GPUDevice>(), Tinput_v, block_size_,
++        functor::DepthToSpaceOpFunctor<Device, int32, FORMAT_NCHW> functor;
++        functor(context->eigen_device<Device>(), Tinput_v, block_size_,
+                 Toutput_v);
+         return;
+       } else if (data_format_ == FORMAT_NCHW) {
+-        functor::DepthToSpaceOpFunctor<GPUDevice, T, FORMAT_NCHW> functor;
+-        functor(context->eigen_device<GPUDevice>(), Tinput, block_size_,
++        functor::DepthToSpaceOpFunctor<Device, T, FORMAT_NCHW> functor;
++        functor(context->eigen_device<Device>(), Tinput, block_size_,
+                 Toutput);
+         return;
+       }
+@@ -173,6 +173,15 @@
+     }
+   }
+ };
++#ifdef WIN32
++template <typename T>
++struct DepthToSpaceOpFunctor<CPUDevice, T, FORMAT_NCHW> {
++  void operator()(const CPUDevice& d, typename TTypes<T, 4>::ConstTensor input,
++                  int block_size, typename TTypes<T, 4>::Tensor output) {
++    LOG(FATAL) << "dummy implementation to make debug build compile";
++  }
++};
++#endif
+ }  // namespace functor
+ 
+ #define REGISTER(type)                                                   \
+
+diff --git a/tensorflow/core/kernels/spacetodepth_op.cc b/tensorflow/core/kernels/spacetodepth_op.cc
+--- a/tensorflow/core/kernels/spacetodepth_op.cc
++++ b/tensorflow/core/kernels/spacetodepth_op.cc
+@@ -132,18 +132,18 @@
+         // NCHW_VECT_C with 4 x qint8 can be treated as NCHW int32.
+         auto Tinput_v = input.template reinterpret_last_dimension<int32, 4>();
+         auto Toutput_v = outputs_tensor->reinterpret_last_dimension<int32, 4>();
+-        functor::SpaceToDepthOpFunctor<GPUDevice, int32, FORMAT_NCHW> functor;
+-        functor(context->eigen_device<GPUDevice>(), Tinput_v, block_size_,
++        functor::SpaceToDepthOpFunctor<Device, int32, FORMAT_NCHW> functor;
++        functor(context->eigen_device<Device>(), Tinput_v, block_size_,
+                 Toutput_v);
+       } else if (data_format_ == FORMAT_NCHW) {
+         CHECK((std::is_same<T, RT>::value));
+-        functor::SpaceToDepthOpFunctor<GPUDevice, RT, FORMAT_NCHW> functor;
+-        functor(context->eigen_device<GPUDevice>(), input.tensor<RT, 4>(),
++        functor::SpaceToDepthOpFunctor<Device, RT, FORMAT_NCHW> functor;
++        functor(context->eigen_device<Device>(), input.tensor<RT, 4>(),
+                 block_size_, outputs_tensor->tensor<RT, 4>());
+       } else {
+         CHECK((std::is_same<T, RT>::value));
+-        functor::SpaceToDepthOpFunctor<GPUDevice, RT, FORMAT_NHWC> functor;
+-        functor(context->eigen_device<GPUDevice>(), input.tensor<RT, 4>(),
++        functor::SpaceToDepthOpFunctor<Device, RT, FORMAT_NHWC> functor;
++        functor(context->eigen_device<Device>(), input.tensor<RT, 4>(),
+                 block_size_, outputs_tensor->tensor<RT, 4>());
+       }
+     } else {
+@@ -188,6 +188,15 @@
+     }
+   }
+ };
++#ifdef WIN32
++template <typename T>
++struct SpaceToDepthOpFunctor<CPUDevice, T, FORMAT_NCHW> {
++  void operator()(const CPUDevice& d, typename TTypes<T, 4>::ConstTensor input,
++                  int block_size, typename TTypes<T, 4>::Tensor output) {
++    LOG(FATAL) << "dummy implementation to make debug build compile";
++  }
++};
++#endif
+ }  // namespace functor
+ 
+ #define REGISTER(type)                                                   \