Merge RFC4 (#1865)

This commit is the result of the squashing of rfc4_dev branch in a single
commit. It implements mostly RFC 4 related work.

* Grid handling:
  - remove obsolete and presumably unfinished implementation of grid catalog functionality
  - all grid functionality is in grids.cpp/.hpp
  - vertical and horizontal grid shift: rework to no longer load whole grid into memory
  - remove hgrids and vgrids member from PJ structure, and store them in hgridshift/vgridshift/deformation structures
  - build systems: add optional libtiff dependency. Must be explicitly disabled if not desired
  - add support for horizontal and vertical grids in GeoTIFF, if libtiff is available
  - add GenericShiftGridSet and GenericShiftGrid classes, relying on TIFF grids, that can be used for generic purpose grid-based adjustment
  - add a +proj=xyzgridshift method to perform geocentric translation by grid. Used for French NTF to RGF93 transformation using gr3df97a.tif grid
  - deformation: add support for +grids= for GeoTIFF grids
  - horizontal grid shift: fix failures on points slightly outside a subgrid (fixes #209)

* File management:
  - add a filemanager.cpp/.hpp to deal with file related work
  - test for legacy proj_api.h fileapi
  - proj.h: add proj_context_set_fileapi() and proj_context_set_sqlite3_vfs_name() (fixes #866)
  - add capability to read resource files from the user writable directory

* Network access:
  - build systems: add optional curl dependency
  - add a curl-based default implementation for network related functionality
  - proj.h: add C API to control network functionality, and optionaly provide network callbacks
  - add data/proj.ini with default settings
  - add a SQLite3 local cache of downloaded chunks
  - add proj_is_download_needed() and proj_download_file()

* Use Win32 Unicode APIs and expect all strings to be UTF-8 (fixes #1765)
  For backward compatibility, if PROJ_LIB content is found to be not UTF-8 or
  pointing to a non existing directory, then an attempt at interpretating it
  in the ANSI page encoding is done.
  proj_context_set_search_paths() now assumes strings to be in UTF-8, and
  functions returning paths will also return values in UTF-8.

5 files changed, 127 insertions, 269 deletions

diff --git a/docs/source/usage/differences.rst b/docs/source/usage/differences.rst
index e85ba039..be163176 100644
--- a/docs/source/usage/differences.rst
+++ b/docs/source/usage/differences.rst
@@ -139,3 +139,9 @@ cs2cs
 
 Removed ``-ld`` option from application, since it promoted use of deprecated
 paramters like ``+towgs`` and ``+datum``.
+
+UTF-8 adoption
+--------------
+
+The value of all path, filenames passed to PROJ through function calls, PROJ
+strings or environment variables should be encoded in UTF-8.
diff --git a/docs/source/usage/environmentvars.rst b/docs/source/usage/environmentvars.rst
index 457432a0..31a74e9a 100644
--- a/docs/source/usage/environmentvars.rst
+++ b/docs/source/usage/environmentvars.rst
@@ -56,3 +56,21 @@ done by setting the variable with no content::
     Set the debug level of PROJ. The default debug level is zero, which results
     in no debug output when using PROJ. A number from 1-3, whit 3 being the most
     verbose setting.
+
+.. envvar:: PROJ_NETWORK
+
+    .. versionadded:: 7.0.0
+
+    If set to ON, enable the capability to use remote grids stored on CDN
+    (Content Delivery Network) storage, when grids are not available locally.
+    Alternatively, the :c:func:`proj_context_set_enable_network` function can
+    be used.
+
+.. envvar:: PROJ_NETWORK_ENDPOINT
+
+    .. versionadded:: 7.0.0
+
+    Define the endpoint of the CDN storage. Normally defined through the proj.ini
+    configuration file locale in :envvar:`PROJ_LIB`.
+    Alternatively, the :c:func:`proj_context_set_url_endpoint` function can
+    be used.
diff --git a/docs/source/usage/index.rst b/docs/source/usage/index.rst
index c31c6dce..a074b18b 100644
--- a/docs/source/usage/index.rst
+++ b/docs/source/usage/index.rst
@@ -17,5 +17,4 @@ command line applications or the C API that is a part of the software package.
    transformation
    environmentvars
    differences
-   projjson
-
+   network
diff --git a/docs/source/usage/network.rst b/docs/source/usage/network.rst
new file mode 100644
index 00000000..295232b0
--- /dev/null
+++ b/docs/source/usage/network.rst
@@ -0,0 +1,102 @@
+.. _network:
+
+================================================================================
+Network capabilities
+================================================================================
+
+.. versionadded:: 7.0
+
+PROJ 7.0 has introduced, per :ref:`rfc4`, the capability to work with grid files
+that are not installed on the local machine where PROJ is executed.
+
+This enables to transparently download the parts of grids that are needed to
+perform a coordinate transformation.
+
+.. _cdn_grids:
+
+CDN of GeoTIFF grids
+--------------------
+
+Files are accessed by default through a CDN (Content Delivery Network),
+accessible through https://cdn.proj.org, that contains :ref:`geodetictiffgrids`
+datasets which are mirrored and managed by the
+https://github.com/OSGeo/proj-datumgrid-geotiff/ GitHub project.
+Files in the CDN are designed to be used by PROJ 7 or later, but any software
+project wishing to use the CDN for shifting support are encouraged to
+participate in the project and leverage the CDN. 
+
+How to enable network capabilities ?
+------------------------------------
+
+This capability assumes that PROJ has been build against `libcurl`, and that
+the user authorizes network access.
+
+Authorizing network access can be done in multiple ways:
+
+  - enabling / uncommenting the ``network = on`` line of :ref:`proj-ini`
+  - definiting the :envvar:`PROJ_NETWORK` environment variable to ON
+  - or using the :cpp:func:`proj_context_set_enable_network` with a
+    ``enabled`` = TRUE value.
+
+.. note::
+
+    Instead of using the `libcurl` implementation, an application using the PROJ
+    API can supply its own network implementation through C function callbacks
+    with :cpp:func:`proj_context_set_network_callbacks`. Enabling network use
+    must still be done with one of the above mentionned method.
+
+Setting endpoint
+----------------
+
+When this is enabled, and a grid is not found in the various locations where
+:ref:`resource files are looked for <resource_file_paths>`, PROJ will then
+attempt at loading the file from a remote server, which defaults to
+https://cdn.proj.org in :ref:`proj-ini`. This location can be changed with
+the :envvar:`PROJ_NETWORK_ENDPOINT` environment variable or with
+:cpp:func:`proj_context_set_url_endpoint`.
+
+Caching
+-------
+
+To avoid repeated access to network, a local cache of downloaded chunks of grids
+is implemented as SQLite3 database, ``cache.db``, stored in the
+:ref:`PROJ user writable directory <user_writable_directory>`.
+
+This local caching is enabled by default (can be changed in :ref:`proj-ini` or
+with :cpp:func:`proj_grid_cache_set_enable`). The default maximum size of the
+cache is 300 MB, which is more than half of the total size of grids available,
+at time of writing. This size can also be customized in :ref:`proj-ini` or
+with :cpp:func:`proj_grid_cache_set_max_size`
+
+Download API
+------------
+
+When on-demand loading of grid is not desirable, the PROJ API also offers the
+capability to download whole grids in the
+:ref:`PROJ user writable directory <user_writable_directory>` by using the
+:cpp:func:`proj_is_download_needed` and :cpp:func:`proj_download_file` functions.
+
+
+Mirroring
+---------
+
+If you are able, you are encouraged to mirror the grids via AWS S3 command line:
+
+::
+
+    aws s3 sync s3://cdn.proj.org .
+
+If direct S3 access is not possible, you can also use wget to locally mirror the
+data:
+
+::
+
+    wget --mirror https://cdn.proj.org/
+
+Acknowledgments
+---------------
+
+The s3://cdn.proj.org bucket is hosted by the
+`Amazon Public Datasets program <https://aws.amazon.com/opendata/public-datasets/>`_.
+CDN services are provided by the AWS Public Dataset team via
+`CloudFront <https://aws.amazon.com/cloudfront/>`_
diff --git a/docs/source/usage/projjson.rst b/docs/source/usage/projjson.rst
deleted file mode 100644
index 6bcce585..00000000
--- a/docs/source/usage/projjson.rst
+++ /dev/null
@@ -1,267 +0,0 @@
-.. _projjson:
-
-================================================================================
-PROJJSON
-================================================================================
-
-PROJJSON is a JSON encoding of
-`WKT2:2019 / ISO-19162:2019 <http://docs.opengeospatial.org/is/18-010r7/18-010r7.html>`_,
-which itself implements the model of
-`OGC Topic 2: Referencing by coordinates <http://docs.opengeospatial.org/as/18-005r4/18-005r4.html>`_.
-Apart from the difference of encodings, the semantics is intended to be exactly
-the same as WKT2:2019.
-
-PROJJSON is available as input and output of PROJ since PROJ 6.2.
-
-The current version is 0.2.
-
-Schema
-------
-
-A JSON schema of its grammar is available at
-https://proj.org/schemas/v0.2/projjson.schema.json
-
-History
--------
-
-* v0.2: addition of geoid_model in VerticalCRS object.
-* v0.1: initial version for PROJ 6.2
-
-Content
--------
-
-The high level objects are:
-
-* Coordinate Reference Systems (CRS):
-
-  - Common ones:
-
-    + ``GeographicCRS``
-    + ``GeodeticCRS``
-    + ``ProjectedCRS``
-    + ``CompoundCRS``
-    + ``BoundCRS``
-
-  - More esoteric ones:
-
-    + ``VerticalCRS``
-    + ``EngineeringCRS``
-    + ``TemporalCRS``
-    + ``ParametricCRS``
-    + ``DerivedGeographicCRS``
-    + ``DerivedGeodeticCRS``
-    + ``DerivedProjectedCRS``
-    + ``DerivedVerticalCRS``
-    + ``DerivedEngineeringCRS``
-    + ``DerivedTemporalCRS``
-    + ``DerivedParametricCRS``
-
-* Coordinate operations:
-
-  - ``Transformation``
-  - ``Conversion``
-  - ``ConcatenatedOperation``
-
-* Others:
-
-  - ``PrimeMeridian``
-  - ``Ellipsoid``
-  - ``Datum``
-  - ``DatumEnsemble``
-
-Examples
---------
-
-GeographicCRS
-+++++++++++++
-
-The following invokation
-
-::
-
-    projinfo EPSG:4326 -o PROJJSON -q
-
-will output:
-
-.. code-block:: json
-
-    {
-        "$schema": "https://proj.org/schemas/v0.1/projjson.schema.json",
-        "type": "GeographicCRS",
-        "name": "WGS 84",
-        "datum": {
-            "type": "GeodeticReferenceFrame",
-            "name": "World Geodetic System 1984",
-            "ellipsoid": {
-                "name": "WGS 84",
-                "semi_major_axis": 6378137,
-                "inverse_flattening": 298.257223563
-            }
-        },
-        "coordinate_system": {
-            "subtype": "ellipsoidal",
-            "axis": [
-            {
-                "name": "Geodetic latitude",
-                "abbreviation": "Lat",
-                "direction": "north",
-                "unit": "degree"
-            },
-            {
-                "name": "Geodetic longitude",
-                "abbreviation": "Lon",
-                "direction": "east",
-                "unit": "degree"
-            }
-            ]
-        },
-        "area": "World",
-        "bbox": {
-            "south_latitude": -90,
-            "west_longitude": -180,
-            "north_latitude": 90,
-            "east_longitude": 180
-        },
-        "id": {
-            "authority": "EPSG",
-            "code": 4326
-        }
-    }
-
-
-ProjectedCRS
-++++++++++++
-
-The following invokation
-
-::
-
-    projinfo EPSG:32631 -o PROJJSON -q
-
-will output:
-
-.. code-block:: json
-
-    {
-        "$schema": "https://proj.org/schemas/v0.1/projjson.schema.json",
-        "type": "ProjectedCRS",
-        "name": "WGS 84 / UTM zone 31N",
-        "base_crs": {
-            "name": "WGS 84",
-            "datum": {
-                "type": "GeodeticReferenceFrame",
-                "name": "World Geodetic System 1984",
-                "ellipsoid": {
-                    "name": "WGS 84",
-                    "semi_major_axis": 6378137,
-                    "inverse_flattening": 298.257223563
-                }
-            },
-            "coordinate_system": {
-                "subtype": "ellipsoidal",
-                "axis": [
-                {
-                    "name": "Geodetic latitude",
-                    "abbreviation": "Lat",
-                    "direction": "north",
-                    "unit": "degree"
-                },
-                {
-                    "name": "Geodetic longitude",
-                    "abbreviation": "Lon",
-                    "direction": "east",
-                    "unit": "degree"
-                }
-                ]
-            },
-            "id": {
-                "authority": "EPSG",
-                "code": 4326
-            }
-        },
-        "conversion": {
-            "name": "UTM zone 31N",
-            "method": {
-                "name": "Transverse Mercator",
-                "id": {
-                    "authority": "EPSG",
-                    "code": 9807
-                }
-            },
-            "parameters": [
-            {
-                "name": "Latitude of natural origin",
-                "value": 0,
-                "unit": "degree",
-                "id": {
-                "authority": "EPSG",
-                "code": 8801
-                }
-            },
-            {
-                "name": "Longitude of natural origin",
-                "value": 3,
-                "unit": "degree",
-                "id": {
-                "authority": "EPSG",
-                "code": 8802
-                }
-            },
-            {
-                "name": "Scale factor at natural origin",
-                "value": 0.9996,
-                "unit": "unity",
-                "id": {
-                "authority": "EPSG",
-                "code": 8805
-                }
-            },
-            {
-                "name": "False easting",
-                "value": 500000,
-                "unit": "metre",
-                "id": {
-                "authority": "EPSG",
-                "code": 8806
-                }
-            },
-            {
-                "name": "False northing",
-                "value": 0,
-                "unit": "metre",
-                "id": {
-                "authority": "EPSG",
-                "code": 8807
-                }
-            }
-            ]
-        },
-        "coordinate_system": {
-            "subtype": "Cartesian",
-            "axis": [
-            {
-                "name": "Easting",
-                "abbreviation": "E",
-                "direction": "east",
-                "unit": "metre"
-            },
-            {
-                "name": "Northing",
-                "abbreviation": "N",
-                "direction": "north",
-                "unit": "metre"
-            }
-            ]
-        },
-        "area": "World - N hemisphere - 0°E to 6°E - by country",
-        "bbox": {
-            "south_latitude": 0,
-            "west_longitude": 0,
-            "north_latitude": 84,
-            "east_longitude": 6
-        },
-        "id": {
-            "authority": "EPSG",
-            "code": 32631
-        }
-    }