From a3f43744feec86272fe532124679d3a013ef9a8c Mon Sep 17 00:00:00 2001 From: PROJ deploybot Date: Tue, 22 Mar 2022 20:00:06 +0000 Subject: update with results of commit https://github.com/OSGeo/PROJ/commit/53c07a8bd211b7aee4bc07a9c6726005504b7181 --- community/channels.html | 175 +++++++ community/code_contributions.html | 301 +++++++++++ community/code_of_conduct.html | 222 ++++++++ community/contributing.html | 301 +++++++++++ community/index.html | 160 ++++++ community/rfc/index.html | 161 ++++++ community/rfc/rfc-1.html | 325 ++++++++++++ community/rfc/rfc-2.html | 1032 +++++++++++++++++++++++++++++++++++++ community/rfc/rfc-3.html | 291 +++++++++++ community/rfc/rfc-4.html | 881 +++++++++++++++++++++++++++++++ community/rfc/rfc-5.html | 280 ++++++++++ community/rfc/rfc-6.html | 474 +++++++++++++++++ community/rfc/rfc-7.html | 294 +++++++++++ 13 files changed, 4897 insertions(+) create mode 100644 community/channels.html create mode 100644 community/code_contributions.html create mode 100644 community/code_of_conduct.html create mode 100644 community/contributing.html create mode 100644 community/index.html create mode 100644 community/rfc/index.html create mode 100644 community/rfc/rfc-1.html create mode 100644 community/rfc/rfc-2.html create mode 100644 community/rfc/rfc-3.html create mode 100644 community/rfc/rfc-4.html create mode 100644 community/rfc/rfc-5.html create mode 100644 community/rfc/rfc-6.html create mode 100644 community/rfc/rfc-7.html (limited to 'community') diff --git a/community/channels.html b/community/channels.html new file mode 100644 index 00000000..7ba9cf6a --- /dev/null +++ b/community/channels.html @@ -0,0 +1,175 @@ + + + + + + + Communication channels — PROJ 9.0.0 documentation + + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Communication channels

+
+

Mailing list

+

Users and developers of the library are using the mailing list to discuss all +things related to PROJ. The mailing list is the primary forum for asking for +help with use of PROJ. The mailing list is also used for announcements, discussions +about the development of the library and from time to time interesting discussions +on geodesy appear as well. You are more than welcome to join in on the discussions!

+

The PROJ mailing list can be found at https://lists.osgeo.org/mailman/listinfo/proj

+
+
+

GitHub

+

GitHub is the development platform we use for collaborating on the PROJ code. +We use GitHub to keep track of the changes in the code and to index bug reports +and feature requests. We are happy to take contributions in any form, either +as code, bug reports, documentation or feature requests. See Contributing +for more info on how you can help improve PROJ.

+

The PROJ GitHub page can be found at https://github.com/OSGeo/PROJ

+
+

Note

+

The issue tracker on GitHub is only meant to keep track of bugs, feature +request and other things related to the development of PROJ. Please ask +your questions about the use of PROJ on the mailing list instead.

+
+
+
+

Gitter

+

Gitter is the instant messaging alternative to the mailing list. PROJ has a +room under the OSGeo organization. Most of the core developers stop by from +time to time for an informal chat. You are more than welcome to join the +discussion.

+

The Gitter room can be found at https://gitter.im/OSGeo/proj.4

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/community/code_contributions.html b/community/code_contributions.html new file mode 100644 index 00000000..406f7102 --- /dev/null +++ b/community/code_contributions.html @@ -0,0 +1,301 @@ + + + + + + + Guidelines for PROJ code contributors — PROJ 9.0.0 documentation + + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Guidelines for PROJ code contributors

+

This is a guide for PROJ, casual or regular, code contributors.

+
+

Code contributions.

+

Code contributions can be either bug fixes or new features. The process +is the same for both, so they will be discussed together in this +section.

+
+

Making Changes

+
    +

  • Create a topic branch from where you want to base your work.

    • +

  • You usually should base your topic branch off of the master branch.

    • +

  • To quickly create a topic branch: git checkout -b my-topic-branch

    • +

  • Make commits of logical units.

    • +

  • Check for unnecessary whitespace with git diff --check before +committing.

    • +

  • Make sure your commit messages are in the proper +format.

    • +

  • Make sure you have added the necessary tests for your changes.

    • +

  • Make sure that all tests pass

    • +
+
+
+

Submitting Changes

+
    +

  • Push your changes to a topic branch in your fork of the repository.

    • +

  • Submit a pull request to the PROJ repository in the OSGeo +organization.

    • +

  • If your pull request fixes/references an issue, include that issue +number in the pull request. For example:

    • +
+
Wiz the bang
+
+Fixes #123.
+
+
+
    +

  • PROJ developers will look at your patch and take an appropriate +action.

    • +
+
+
+

Coding conventions

+
+

Programming language

+

PROJ was originally developed in ANSI C. Today PROJ is mostly developed in C++11, +with a few parts of the code base still being C. Most of the older parts of the +code base is effectively C with a few modifications so that it compiles better as +C++.

+
+
+

Coding style

+

The parts of the code base that has started its life as C++ is formatted with +clang-format using the script scripts/reformat_cpp.sh. This is mostly +contained to the code in src/iso19111/ but a few other .cpp-files are +covered as well.

+

For the rest of the code base, which has its origin in C, we don’t enforce any +particular coding style, but please try to keep it as simple as possible. If +improving existing code, please try to conform with the style of the locally +surrounding code.

+
+
+

Whitespace

+

Throughout the PROJ code base you will see differing whitespace use. +The general rule is to keep whitespace in whatever form it is in the +file you are currently editing. If the file has a mix of tabs and space +please convert the tabs to space in a separate commit before making any +other changes. This makes it a lot easier to see the changes in diffs +when evaluating the changed code. New files should use spaces as +whitespace.

+
+
+
+
+

Tools

+
+

Reformatting C++ code

+

The script in scripts/reformat_cpp.sh will reformat C++ code in accordance +to the project preference.

+

If you are writing a new .cpp-file it should be added to the list in the +reformatting script.

+
+
+

cppcheck static analyzer

+

You can run locally scripts/cppcheck.sh that is a wrapper script around the +cppcheck utility. This tool is used as part of the quality control of the code.

+

cppcheck can have false positives. In general, it is preferable to rework the +code a bit to make it more ‘obvious’ and avoid those false positives. When not +possible, you can add a comment in the code like

+
/* cppcheck-suppress duplicateBreak */
+
+
+

in the preceding line. Replace +duplicateBreak with the actual name of the violated rule emitted by cppcheck.

+
+
+

Clang Static Analyzer (CSA)

+

CSA is run by a GitHub Actions workflow. You may also run it locally.

+

Preliminary step: install clang. For example:

+
wget https://releases.llvm.org/9.0.0/clang+llvm-9.0.0-x86_64-linux-gnu-ubuntu-18.04.tar.xz
+tar xJf clang+llvm-9.0.0-x86_64-linux-gnu-ubuntu-18.04.tar.xz
+mv clang+llvm-9.0.0-x86_64-linux-gnu-ubuntu-18.04 clang+llvm-9
+export PATH=$PWD/clang+llvm-9/bin:$PATH
+
+
+

Configure PROJ with the scan-build utility of clang:

+
+
::

mkdir csa_build +cd csa_build +scan-build cmake ..

+
+
+

Build using scan-build:

+
scan-build make [-j8]
+
+
+

If CSA finds errors, they will be emitted during the build. And in which case, +at the end of the build process, scan-build will emit a warning message +indicating errors have been found and how to display the error report. This +is with something like

+
scan-view /tmp/scan-build-2021-03-15-121416-17476-1
+
+
+

This will open a web browser with the interactive report.

+

CSA may also have false positives. In general, this happens when the code is +non-trivial / makes assumptions that hard to check at first sight. You will +need to add extra checks or rework it a bit to make it more “obvious” for CSA. +This will also help humans reading your code !

+
+
+

Typo detection and fixes

+

Run scripts/fix_typos.sh

+
+
+

Include What You Use (IWYU)

+

Managing C includes is a pain. IWYU makes updating headers a bit +easier. IWYU scans the code for functions that are called and makes +sure that the headers for all those functions are present and in +sorted order. However, you cannot blindly apply IWYU to PROJ. It +does not understand ifdefs, other platforms, or the order requirements +of PROJ internal headers. So the way to use it is to run it on a copy +of the source and merge in only the changes that make sense. +Additions of standard headers should always be safe to merge. The +rest require careful evaluation. See the IWYU documentation for +motivation and details.

+

IWYU docs

+
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/community/code_of_conduct.html b/community/code_of_conduct.html new file mode 100644 index 00000000..e5370f4a --- /dev/null +++ b/community/code_of_conduct.html @@ -0,0 +1,222 @@ + + + + + + + Code of Conduct — PROJ 9.0.0 documentation + + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Code of Conduct

+

The PROJ project has adopted the +Contributor Covenant Code of Conduct. +Everyone who participates in the PROJ community is expected to follow the +code of conduct as written below.

+
+

Our Pledge

+

In the interest of fostering an open and welcoming environment, we as +contributors and maintainers pledge to make participation in our project and +our community a harassment-free experience for everyone, regardless of age, body +size, disability, ethnicity, sex characteristics, gender identity and expression, +level of experience, education, socio-economic status, nationality, personal +appearance, race, religion, or sexual identity and orientation.

+
+
+

Our Standards

+

Examples of behavior that contributes to creating a positive environment +include:

+
    +

  • Using welcoming and inclusive language

    • +

  • Being respectful of differing viewpoints and experiences

    • +

  • Gracefully accepting constructive criticism

    • +

  • Focusing on what is best for the community

    • +

  • Showing empathy towards other community members

    • +
+

Examples of unacceptable behavior by participants include:

+
    +

  • The use of sexualized language or imagery and unwelcome sexual attention or +advances

    • +

  • Trolling, insulting/derogatory comments, and personal or political attacks

    • +

  • Public or private harassment

    • +

  • Publishing others’ private information, such as a physical or electronic +address, without explicit permission

    • +

  • Other conduct which could reasonably be considered inappropriate in a +professional setting

    • +
+
+
+

Our Responsibilities

+

Project maintainers are responsible for clarifying the standards of acceptable +behavior and are expected to take appropriate and fair corrective action in +response to any instances of unacceptable behavior.

+

Project maintainers have the right and responsibility to remove, edit, or +reject comments, commits, code, wiki edits, issues, and other contributions +that are not aligned to this Code of Conduct, or to ban temporarily or +permanently any contributor for other behaviors that they deem inappropriate, +threatening, offensive, or harmful.

+
+
+

Scope

+

This Code of Conduct applies within all project spaces, and it also applies when +an individual is representing the project or its community in public spaces. +Examples of representing a project or community include using an official +project e-mail address, posting via an official social media account, or acting +as an appointed representative at an online or offline event. Representation of +a project may be further defined and clarified by project maintainers.

+
+
+

Enforcement

+

Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported by contacting the project team at kristianevers@gmail.com. +All complaints will be reviewed and investigated and will result in a response that +is deemed necessary and appropriate to the circumstances. The project team is +obligated to maintain confidentiality with regard to the reporter of an incident. +Further details of specific enforcement policies may be posted separately.

+

Project maintainers who do not follow or enforce the Code of Conduct in good +faith may face temporary or permanent repercussions as determined by other +members of the project’s leadership.

+
+
+

Attribution

+

This Code of Conduct is adapted from the +Contributor Covenant, version 1.4, +available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html

+

For answers to common questions about this code of conduct, see +https://www.contributor-covenant.org/faq

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/community/contributing.html b/community/contributing.html new file mode 100644 index 00000000..11a214c4 --- /dev/null +++ b/community/contributing.html @@ -0,0 +1,301 @@ + + + + + + + Contributing — PROJ 9.0.0 documentation + + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Contributing

+

PROJ has a wide and varied user base. Some are highly skilled +geodesists with a deep knowledge of map projections and reference +systems, some are GIS software developers and others are GIS users. All +users, regardless of the profession or skill level, has the ability to +contribute to PROJ. Here’s a few suggestion on how:

+
    +

  • Help PROJ-users that is less experienced than yourself.

    • +

  • Write a bug report

    • +

  • Request a new feature

    • +

  • Write documentation for your favorite map projection

    • +

  • Fix a bug

    • +

  • Implement a new feature

    • +
+

In the following sections you can find some guidelines on how to +contribute. As PROJ is managed on GitHub most contributions require +that you have a GitHub account. Familiarity with +issues and the GitHub +Flow is an advantage.

+
+

Help a fellow PROJ user

+

The main forum for support for PROJ is the mailing list. You can +subscribe to the mailing list +here and read the +archive here.

+

If you have questions about the usage of PROJ the mailing list is also +the place to go. Please do not use the GitHub issue tracker as a +support forum. Your question is much more likely to be answered on the +mailing list, as many more people follow that than the issue tracker.

+
+
+

Adding bug reports

+

Bug reports are handled in the issue +tracker on PROJ’s home on +GitHub. Writing a good bug report is not easy. But fixing a poorly +documented bug is not easy either, so please put in the effort it takes +to create a thorough bug report.

+

A good bug report includes at least:

+
    +

  • A title that quickly explains the problem

    • +

  • A description of the problem and how it can be reproduced

    • +

  • Version of PROJ being used

    • +

  • Version numbers of any other relevant software being used, e.g. +operating system

    • +

  • A description of what already has been done to solve the problem

    • +
+

The more information that is given up front, the more likely it is that +a developer will find interest in solving the problem. You will probably +get follow-up questions after submitting a bug report. Please answer +them in a timely manner if you have an interest in getting the issue +solved.

+

Finally, please only submit bug reports that are actually related to +PROJ. If the issue materializes in software that uses PROJ it is +likely a problem with that particular software. Make sure that it +actually is a PROJ problem before you submit an issue. If you can +reproduce the problem only by using tools from PROJ it is definitely a +problem with PROJ.

+
+
+

Feature requests

+

Got an idea for a new feature in PROJ? Submit a thorough description +of the new feature in the issue +tracker. Please include any +technical documents that can help the developer make the new feature a +reality. An example of this could be a publicly available academic paper +that describes a new projection. Also, including a numerical test case +will make it much easier to verify that an implementation of your +requested feature actually works as you expect.

+

Note that not all feature requests are accepted.

+
+
+

Write documentation

+

PROJ is in dire need of better documentation. Any contributions of +documentation are greatly appreciated. The PROJ documentation is +available on proj.org. The website is generated +with Sphinx. Contributions to +the documentation should be made as Pull +Requests on GitHub.

+

If you intend to document one of PROJ’s supported projections please +use the Mercator projection +as a template.

+
+
+

Code contributions

+

See Code contributions

+
+

Legalese

+

Committers are the front line gatekeepers to keep the code base clear of +improperly contributed code. It is important to the PROJ users, +developers and the OSGeo foundation to avoid contributing any code to +the project without it being clearly licensed under the project license.

+

Generally speaking the key issues are that those providing code to be +included in the repository understand that the code will be released +under the MIT/X license, and that the person providing the code has the +right to contribute the code. For the committer themselves understanding +about the license is hopefully clear. For other contributors, the +committer should verify the understanding unless the committer is very +comfortable that the contributor understands the license (for instance +frequent contributors).

+

If the contribution was developed on behalf of an employer (on work +time, as part of a work project, etc) then it is important that an +appropriate representative of the employer understand that the code will +be contributed under the MIT/X license. The arrangement should be +cleared with an authorized supervisor/manager, etc.

+

The code should be developed by the contributor, or the code should be +from a source which can be rightfully contributed such as from the +public domain, or from an open source project under a compatible +license.

+

All unusual situations need to be discussed and/or documented.

+

Committers should adhere to the following guidelines, and may be +personally legally liable for improperly contributing code to the source +repository:

+
    +

  • Make sure the contributor (and possibly employer) is aware of the +contribution terms.

    • +

  • Code coming from a source other than the contributor (such as adapted +from another project) should be clearly marked as to the original +source, copyright holders, license terms and so forth. This +information can be in the file headers, but should also be added to +the project licensing file if not exactly matching normal project +licensing (COPYING).

    • +

  • Existing copyright headers and license text should never be stripped +from a file. If a copyright holder wishes to give up copyright they +must do so in writing to the foundation before copyright messages are +removed. If license terms are changed it has to be by agreement +(written in email is ok) of the copyright holders.

    • +

  • Code with licenses requiring credit, or disclosure to users should be +added to COPYING.

    • +

  • When substantial contributions are added to a file (such as +substantial patches) the author/contributor should be added to the +list of copyright holders for the file.

    • +

  • If there is uncertainty about whether a change is proper to +contribute to the code base, please seek more information from the +project steering committee, or the foundation legal counsel.

    • +
+
+
+
+

Additional Resources

+ +
+
+

Acknowledgements

+

The code contribution section of this CONTRIBUTING file is inspired by +PDAL’s +and the legalese section is modified from GDAL committer +guidelines

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/community/index.html b/community/index.html new file mode 100644 index 00000000..5df7df95 --- /dev/null +++ b/community/index.html @@ -0,0 +1,160 @@ + + + + + + + Community — PROJ 9.0.0 documentation + + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

Community

+

The PROJ community is what makes the software stand out from its competitors. +PROJ is used and developed by group of very enthusiastic, knowledgeable and +friendly people. Whether you are a first time user of PROJ or a long-time +contributor the community is always very welcoming.

+
+ +
+
+

Conference

+FOSS4G 2021 +

FOSS4G 2021 is the leading annual conference for +free and open source geospatial software. It will include presentations related +to PROJ, and some of the PROJ development community will be attending. It is the +event for those interested in PROJ, other FOSS geospatial technologies and the +community around them. The conference will due to COVID-19 be held in a virtual +setting September 27th - October 2nd, 2021.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/community/rfc/index.html b/community/rfc/index.html new file mode 100644 index 00000000..eb0dd9de --- /dev/null +++ b/community/rfc/index.html @@ -0,0 +1,161 @@ + + + + + + + Request for Comments — PROJ 9.0.0 documentation + + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+ +
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/community/rfc/rfc-1.html b/community/rfc/rfc-1.html new file mode 100644 index 00000000..0bc47530 --- /dev/null +++ b/community/rfc/rfc-1.html @@ -0,0 +1,325 @@ + + + + + + + PROJ RFC 1: Project Committee Guidelines — PROJ 9.0.0 documentation + + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

PROJ RFC 1: Project Committee Guidelines

+
+
Author
+

Frank Warmerdam, Howard Butler

+
+
Contact
+

howard@hobu.co

+
+
Status
+

Passed

+
+
Last Updated
+

2018-06-08

+
+
+
+

Summary

+

This document describes how the PROJ Project Steering Committee (PSC) +determines membership, and makes decisions on all aspects of the +PROJ project - both technical and non-technical.

+

Examples of PSC management responsibilities:

+
    +

  • setting the overall development road map

    • +

  • developing technical standards and policies (e.g. coding standards, +file naming conventions, etc…)

    • +

  • ensuring regular releases (major and maintenance) of PROJ software

    • +

  • reviewing RFC for technical enhancements to the software

    • +

  • project infrastructure (e.g. GitHub, continuous integration hosting options, etc…)

    • +

  • formalization of affiliation with external entities such as OSGeo

    • +

  • setting project priorities, especially with respect to project sponsorship

    • +

  • creation and oversight of specialized sub-committees (e.g. project +infrastructure, training)

    • +
+

In brief the project team votes on proposals on the proj mailing list. Proposals are available for review for at least two days, and a single +veto is sufficient delay progress though ultimately a majority of members can +pass a proposal.

+
+
+

List of PSC Members

+

(up-to-date as of 2018-06)

+ +
+
+

Detailed Process

+
    +

  • Proposals are written up and submitted on the proj mailing list +for discussion and voting, by any interested party, not just +committee members.

    • +

  • Proposals need to be available for review for at least two business +days before a final decision can be made.

    • +

  • Respondents may vote “+1” to indicate support for the proposal and a +willingness to support implementation.

    • +

  • Respondents may vote “-1” to veto a proposal, but must provide clear +reasoning and alternate approaches to resolving the problem within +the two days.

    • +

  • A vote of -0 indicates mild disagreement, but has no effect. A 0 +indicates no opinion. A +0 indicate mild support, but has no +effect.

    • +

  • Anyone may comment on proposals on the list, but only members of the +Project Steering Committee’s votes will be counted.

    • +

  • A proposal will be accepted if it receives +2 (including the +author) and no vetoes (-1).

    • +

  • If a proposal is vetoed, and it cannot be revised to satisfy all +parties, then it can be resubmitted for an override vote in which a +majority of all eligible voters indicating +1 is sufficient to pass it. +Note that this is a majority of all committee members, not just those who +actively vote.

    • +

  • Upon completion of discussion and voting the author should announce +whether they are proceeding (proposal accepted) or are withdrawing +their proposal (vetoed).

    • +

  • The Chair gets a vote.

    • +

  • The Chair is responsible for keeping track of who is a member of the +Project Steering Committee (perhaps as part of a PSC file in CVS).

    • +

  • Addition and removal of members from the committee, as well as selection +of a Chair should be handled as a proposal to the committee.

    • +

  • The Chair adjudicates in cases of disputes about voting.

    • +
+
+

RFC Origin

+

PROJ RFC and Project Steering Committee is derived from similar governance +bodies in both the GDAL and +MapServer software +projects.

+
+
+
+

When is Vote Required?

+
    +

  • Any change to committee membership (new members, removing inactive members)

    • +

  • Changes to project infrastructure (e.g. tool, location or substantive +configuration)

    • +

  • Anything that could cause backward compatibility issues.

    • +

  • Adding substantial amounts of new code.

    • +

  • Changing inter-subsystem APIs, or objects.

    • +

  • Issues of procedure.

    • +

  • When releases should take place.

    • +

  • Anything dealing with relationships with external entities such as OSGeo

    • +

  • Anything that might be controversial.

    • +
+
+
+

Observations

+
    +

  • The Chair is the ultimate adjudicator if things break down.

    • +

  • The absolute majority rule can be used to override an obstructionist +veto, but it is intended that in normal circumstances vetoers need to be +convinced to withdraw their veto. We are trying to reach consensus.

    • +
+
+
+

Committee Membership

+

The PSC is made up of individuals consisting of technical contributors +(e.g. developers) and prominent members of the PROJ user community. +There is no set number of members for the PSC although the initial desire +is to set the membership at 6.

+
+

Adding Members

+

Any member of the proj mailing list may nominate someone for +committee membership at any time. Only existing PSC committee members may +vote on new members. Nominees must receive a majority vote from existing +members to be added to the PSC.

+
+
+

Stepping Down

+

If for any reason a PSC member is not able to fully participate then they +certainly are free to step down. If a member is not active (e.g. no +voting, no IRC or email participation) for a period of two months then +the committee reserves the right to seek nominations to fill that position. +Should that person become active again (hey, it happens) then they would +certainly be welcome, but would require a nomination.

+
+
+
+

Membership Responsibilities

+
+

Guiding Development

+

Members should take an active role guiding the development of new features +they feel passionate about. Once a change request has been accepted +and given a green light to proceed does not mean the members are free of +their obligation. PSC members voting “+1” for a change request are +expected to stay engaged and ensure the change is implemented and +documented in a way that is most beneficial to users. Note that this +applies not only to change requests that affect code, but also those +that affect the web site, technical infrastructure, policies and standards.

+
+
+

Mailing List Participation

+

PSC members are expected to be active on the +proj mailing list, subject to Open Source mailing list +etiquette. Non-developer members of the PSC are not expected to respond +to coding level questions on the developer mailing list, however they +are expected to provide their thoughts and opinions on user level +requirements and compatibility issues when RFC discussions take place.

+
+
+
+

Updates

+

June 2018

+

RFC 1 was ratified by the following members

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/community/rfc/rfc-2.html b/community/rfc/rfc-2.html new file mode 100644 index 00000000..e9dcd2d6 --- /dev/null +++ b/community/rfc/rfc-2.html @@ -0,0 +1,1032 @@ + + + + + + + PROJ RFC 2: Initial integration of “GDAL SRS barn” work — PROJ 9.0.0 documentation + + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

PROJ RFC 2: Initial integration of “GDAL SRS barn” work

+
+
Author
+

Even Rouault

+
+
Contact
+

even.rouault at spatialys.com

+
+
Status
+

Adopted, implemented in PROJ 6.0

+
+
Initial version
+

2018-10-09

+
+
Last Updated
+

2018-10-31

+
+
+
+

Summary

+

This RFC is the result of a first phase of the GDAL Coordinate System Barn Raising +efforts. In its current state, this work mostly consists of:

+
+
    +

  • a C++ implementation of the ISO-19111:2018 / OGC Topic 2 “Referencing by +coordinates” classes to represent Datums, Coordinate systems, CRSs +(Coordinate Reference Systems) and Coordinate Operations.

    • +

  • methods to convert between this C++ modeling and WKT1, WKT2 and PROJ string representations of those objects

    • +

  • management and query of a SQLite3 database of CRS and Coordinate Operation definition

    • +

  • a C API binding part of those capabilities

    • +
+
+
+ +
+

Details

+
+

Structure in packages / namespaces

+

The C++ implementation of the (upcoming) ISO-19111:2018 / OGC Topic 2 “Referencing by +coordinates” classes follows this abstract modeling as much as possible, using +package names as C++ namespaces, abstract classes and method names. A new +BoundCRS class has been added to cover the modeling of the WKT2 BoundCRS +construct, that is a generalization of the WKT1 TOWGS84 concept. It is +strongly recommended to have the ISO-19111 standard open to have an introduction +for the concepts when looking at the code. A few classes have also been +inspired by the GeoAPI

+

The classes are organized into several namespaces:

+
+
    +
  • +
    osgeo::proj::util

    A set of base types from ISO 19103, GeoAPI and other PROJ “technical” +specific classes

    +

    Template optional<T>, classes BaseObject, IComparable, BoxedValue, +ArrayOfBaseObject, PropertyMap, LocalName, NameSpace, GenericName, +NameFactory, CodeList, Exception, InvalidValueTypeException, +UnsupportedOperationException

    +
    +
    +
    • +
  • +
    osgeo::proj::metadata:

    Common classes from ISO 19115 (Metadata) standard

    +

    Classes Citation, GeographicExtent, GeographicBoundingBox, +TemporalExtent, VerticalExtent, Extent, Identifier, PositionalAccuracy,

    +
    +
    +
    • +
  • +
    osgeo::proj::common:

    Common classes: UnitOfMeasure, Measure, Scale, Angle, Length, DateTime, +DateEpoch, IdentifiedObject, ObjectDomain, ObjectUsage

    +
    +
    +
    • +
  • +
    osgeo::proj::cs:

    Coordinate systems and their axis

    +

    Classes AxisDirection, Meridian, CoordinateSystemAxis, CoordinateSystem, +SphericalCS, EllipsoidalCS, VerticalCS, CartesianCS, OrdinalCS, +ParametricCS, TemporalCS, DateTimeTemporalCS, TemporalCountCS, +TemporalMeasureCS

    +
    +
    +
    • +
  • +
    osgeo::proj::datum:

    Datum (the relationship of a coordinate system to the body)

    +

    Classes Ellipsoid, PrimeMeridian, Datum, DatumEnsemble, +GeodeticReferenceFrame, DynamicGeodeticReferenceFrame, +VerticalReferenceFrame, DynamicVerticalReferenceFrame, TemporalDatum, +EngineeringDatum, ParametricDatum

    +
    +
    +
    • +
  • +
    osgeo::proj::crs:

    CRS = coordinate reference system = coordinate system with a datum

    +

    Classes CRS, GeodeticCRS, GeographicCRS, DerivedCRS, ProjectedCRS, +VerticalCRS, CompoundCRS, BoundCRS, TemporalCRS, EngineeringCRS, +ParametricCRS, DerivedGeodeticCRS, DerivedGeographicCRS, +DerivedProjectedCRS, DerivedVerticalCRS

    +
    +
    +
    • +
  • +
    osgeo::proj::operation:

    Coordinate operations (relationship between any two coordinate +reference systems)

    +

    Classes CoordinateOperation, GeneralOperationParameter, +OperationParameter, GeneralParameterValue, ParameterValue, +OperationParameterValue, OperationMethod, InvalidOperation, +SingleOperation, Conversion, Transformation, PointMotionOperation, +ConcatenatedOperation

    +
    +
    +
    • +
  • +
    osgeo::proj::io:

    I/O classes: WKTFormatter, PROJStringFormatter, FormattingException, +ParsingException, IWKTExportable, IPROJStringExportable, WKTNode, +WKTParser, PROJStringParser, DatabaseContext, AuthorityFactory, +FactoryException, NoSuchAuthorityCodeException

    +
    +
    +
    • +
+
+
+
+

What does what?

+

The code to parse WKT and PROJ strings and build ISO-19111 objects is +contained in io.cpp

+

The code to format WKT and PROJ strings from ISO-19111 objects is mostly +contained in the related exportToWKT() and exportToPROJString() methods +overridden in the applicable classes. io.cpp contains the general mechanics +to build such strings.

+

Regarding WKT strings, three variants are handled in import and export:

+
+
    +

  • WKT2_2018: variant corresponding to the upcoming ISO-19162:2018 standard

    • +

  • WKT2_2015: variant corresponding to the current ISO-19162:2015 standard

    • +

  • WKT1_GDAL: variant corresponding to the way GDAL understands the OGC +01-099 and OGC 99-049 standards

    • +
+
+

Regarding PROJ strings, two variants are handled in import and export:

+
+
    +

  • PROJ5: variant used by PROJ >= 5, possibly using pipeline constructs, +and avoiding +towgs84 / +nadgrids legacy constructs. This variant honours +axis order and input/output units. That is the pipeline for the conversion +of EPSG:4326 to EPSG:32631 will assume that the input coordinates are in +latitude, longitude order, with degrees.

    • +

  • PROJ4: variant used by PROJ 4.x

    • +
+
+

The raw query of the proj.db database and the upper level construction of +ISO-19111 objects from the database contents is done in factory.cpp

+
+
+

A few design principles

+

Methods generally take and return xxxNNPtr objects, that is non-null shared +pointers (pointers with internal reference counting). The advantage of this +approach is that the user has not to care about the life-cycle of the +instances (and this makes the code leak-free by design). The only point of +attention is to make sure no reference cycles are made. This is the case for +all classes, except the CoordinateOperation class that point to CRS for +sourceCRS and targetCRS members, whereas DerivedCRS point to a Conversion +instance (which derives from CoordinateOperation). This issue was detected in +the ISO-19111 standard. The solution adopted here is to use std::weak_ptr +in the CoordinateOperation class to avoid the cycle. This design artefact is +transparent to users.

+

Another important design point is that all ISO19111 objects are immutable after +creation, that is they only have getters that do not modify their states. +Consequently they could possibly use in a thread-safe way. There are however +classes like PROJStringFormatter, WKTFormatter, DatabaseContext, AuthorityFactory +and CoordinateOperationContext whose instances are mutable and thus can not be +used by multiple threads at once.

+

Example how to build the EPSG:4326 / WGS84 Geographic2D definition from scratch:

+
auto greenwich = PrimeMeridian::create(
+    util::PropertyMap()
+        .set(metadata::Identifier::CODESPACE_KEY,
+            metadata::Identifier::EPSG)
+        .set(metadata::Identifier::CODE_KEY, 8901)
+        .set(common::IdentifiedObject::NAME_KEY, "Greenwich"),
+    common::Angle(0));
+// actually predefined as PrimeMeridian::GREENWICH constant
+
+auto ellipsoid = Ellipsoid::createFlattenedSphere(
+    util::PropertyMap()
+        .set(metadata::Identifier::CODESPACE_KEY, metadata::Identifier::EPSG)
+        .set(metadata::Identifier::CODE_KEY, 7030)
+        .set(common::IdentifiedObject::NAME_KEY, "WGS 84"),
+    common::Length(6378137),
+    common::Scale(298.257223563));
+// actually predefined as Ellipsoid::WGS84 constant
+
+auto datum = GeodeticReferenceFrame::create(
+    util::PropertyMap()
+        .set(metadata::Identifier::CODESPACE_KEY, metadata::Identifier::EPSG)
+        .set(metadata::Identifier::CODE_KEY, 6326)
+        .set(common::IdentifiedObject::NAME_KEY, "World Geodetic System 1984");
+    ellipsoid
+    util::optional<std::string>(), // anchor
+    greenwich);
+// actually predefined as GeodeticReferenceFrame::EPSG_6326 constant
+
+auto geogCRS = GeographicCRS::create(
+    util::PropertyMap()
+        .set(metadata::Identifier::CODESPACE_KEY, metadata::Identifier::EPSG)
+        .set(metadata::Identifier::CODE_KEY, 4326)
+        .set(common::IdentifiedObject::NAME_KEY, "WGS 84"),
+    datum,
+    cs::EllipsoidalCS::createLatitudeLongitude(scommon::UnitOfMeasure::DEGREE));
+// actually predefined as GeographicCRS::EPSG_4326 constant
+
+
+
+
+

Algorithmic focus

+

On the algorithmic side, a somewhat involved logic is the +CoordinateOperationFactory::createOperations() in coordinateoperation.cpp +that takes a pair of source and target CRS and returns a set of possible +coordinate operations (either single operations like a Conversion or a +Transformation, or concatenated operations). It uses the intrinsic structure +of those objects to create the coordinate operation pipeline. That is, if +going from a ProjectedCRS to another one, by doing first the inverse conversion +from the source ProjectedCRS to its base GeographicCRS, then finding the +appropriate transformation(s) from this base GeographicCRS to the base +GeographicCRS of the target CRS, and then applying the conversion from this +base GeographicCRS to the target ProjectedCRS. At each step, it queries the +database to find if one or several transformations are available. The +resulting coordinate operations are filtered, and sorted, with user provided hints:

+
+
    +

  • desired accuracy

    • +

  • area of use, defined as a bounding box in longitude, latitude space (its +actual CRS does not matter for the intended use)

    • +

  • if no area of use is defined, if and how the area of use of the source +and target CRS should be used. By default, the smallest area of use is +used. The rationale is for example when transforming between a national +ProjectedCRS and a world-scope GeographicCRS to use the are of use of +this ProjectedCRS to select the appropriate datum shifts.

    • +

  • how the area of use of the candidate transformations and the desired area of +use (either explicitly or implicitly defined, as explained above) are +compared. By default, only transformations whose area of use is fully +contained in the desired area of use are selected. It is also possible +to relax this test by specifying that only an intersection test must be used.

    • +

  • whether PROJ transformation grid names should be substituted to the +official names, when a match is found in the grid_alternatives table +of the database. Defaults to true

    • +

  • whether the availability of those grids should be used to filter and sort +the results. By default, the transformations using grids available in the +system will be presented first.

    • +
+
+

The results are sorted, with the most relevant ones appearing first in the +result vector. The criteria used are in that order

+
+
    +

  • grid actual availability: operations referencing grids not available will be +listed after ones with available grids

    • +

  • grid potential availability: operation referencing grids not known at +all in the proj.db will be listed after operations with grids known, but +not available.

    • +

  • known accuracy: operations with unknown accuracies will be listed +after operations with known accuracy

    • +

  • area of use: operations with smaller area of use (the intersection of the +operation area of used with the desired area of use) will be listed after +the ones with larger area of use

    • +

  • accuracy: operations with lower accuracy will be listed after operations +with higher accuracy (caution: lower accuracy actually means a higher numeric +value of the accuracy property, since it is a precision in metre)

    • +
+
+

All those settings can be specified in the CoordinateOperationContext instance +passed to createOperations().

+

An interesting example to understand how those parameters play together is +to use projinfo -s EPSG:4267 -t EPSG:4326 (NAD27 to WGS84 conversions), +and see how specifying desired area of use, spatial criterion, grid availability, +etc. affects the results.

+

The following command currently returns 78 results:

+
projinfo -s EPSG:4267 -t EPSG:4326 --summary --spatial-test intersects
+
+
+

The createOperations() algorithm also does a kind of “CRS routing”. +A typical example is if wanting to transform between CRS A and +CRS B, but no direct transformation is referenced in proj.db between those. +But if there are transformations between A <–> C and B <–> C, then it +is possible to build a concatenated operation A –> C –> B. The typical +example is when C is WGS84, but the implementation is generic and just finds +a common pivot from the database. An example of finding a non-WGS84 pivot is +when searching a transformation between EPSG:4326 and EPSG:6668 (JGD2011 - +Japanese Geodetic Datum 2011), which has no direct transformation registered +in the EPSG database . However there are transformations between those two +CRS and JGD2000 (and also Tokyo datum, but that one involves less accurate +transformations)

+
projinfo -s EPSG:4326 -t EPSG:6668  --grid-check none --bbox 135.42,34.84,142.14,41.58 --summary
+
+Candidate operations found: 7
+unknown id, Inverse of JGD2000 to WGS 84 (1) + JGD2000 to JGD2011 (1), 1.2 m, Japan - northern Honshu
+unknown id, Inverse of JGD2000 to WGS 84 (1) + JGD2000 to JGD2011 (2), 2 m, Japan excluding northern main province
+unknown id, Inverse of Tokyo to WGS 84 (108) + Tokyo to JGD2011 (2), 9.2 m, Japan onshore excluding northern main province
+unknown id, Inverse of Tokyo to WGS 84 (108) + Tokyo to JGD2000 (2) + JGD2000 to JGD2011 (1), 9.4 m, Japan - northern Honshu
+unknown id, Inverse of Tokyo to WGS 84 (2) + Tokyo to JGD2011 (2), 13.2 m, Japan - onshore mainland and adjacent islands
+unknown id, Inverse of Tokyo to WGS 84 (2) + Tokyo to JGD2000 (2) + JGD2000 to JGD2011 (1), 13.4 m, Japan - northern Honshu
+unknown id, Inverse of Tokyo to WGS 84 (1) + Tokyo to JGD2011 (2), 29.2 m, Asia - Japan and South Korea
+
+
+
+
+
+

Code repository

+

The current state of the work can be found in the +iso19111 branch of rouault/proj.4 repository , and is also available as +a GitHub pull request at https://github.com/OSGeo/proj.4/pull/1040

+

Here is a not-so-usable comparison with a fixed snapshot of master branch

+
+
+

Database

+
+

Content

+

The database contains CRS and coordinate operation definitions from the EPSG +database (IOGP’s EPSG Geodetic Parameter Dataset) v9.5.3, +IGNF registry (French National Geographic Institute), ESRI database, as well +as a few customizations.

+
+
+

Building (for PROJ developers creating the database)

+

The building of the database is a several stage process:

+
+

Construct SQL scripts for EPSG

+

The first stage consists in constructing .sql scripts mostly with +CREATE TABLE and INSERT statements to create the database structure and +populate it. There is one .sql file for each database table, populated +with the content of the EPSG database, automatically +generated with the build_db.py script, which processes the PostgreSQL +dumps issued by IOGP. A number of other scripts are dedicated to manual +editing, for example grid_alternatives.sql file that binds official +grid names to PROJ grid names

+
+
+

Concert UTF8 SQL to sqlite3 db

+

The second stage is done automatically by the make process. It pipes the +.sql script, in the right order, to the sqlite3 binary to generate a +first version of the proj.db SQLite3 database.

+
+
+

Add extra registries

+

The third stage consists in creating additional .sql files from the +content of other registries. For that process, we need to bind some +definitions of those registries to those of the EPSG database, to be +able to link to existing objects and detect some boring duplicates. +The ignf.sql file has been generated using +the build_db_create_ignf.py script from the current data/IGNF file +that contains CRS definitions (and implicit transformations to WGS84) +as PROJ.4 strings. +The esri.sql file has been generated using the build_db_from_esri.py +script, from the .csv files in +https://github.com/Esri/projection-engine-db-doc/tree/master/csv

+
+
+

Finalize proj.db

+

The last stage runs make again to incorporate the new .sql files generated +in the previous stage (so the process of building the database involves +a kind of bootstrapping…)

+
+
+
+

Building (for PROJ users)

+

The make process just runs the second stage mentioned above from the .sql +files. The resulting proj.db is currently 5.3 MB large.

+
+
+

Structure

+

The database is structured into the following tables and views. They generally +match a ISO-19111 concept, and is generally close to the general structure of +the EPSG database. Regarding identification of objects, where the EPSG database +only contains a ‘code’ numeric column, the PROJ database identifies objects +with a (auth_name, code) tuple of string values, allowing several registries to +be combined together.

+
    +
  • +
    Technical:
      +

    • authority_list: view enumerating the authorities present in the database. Currently: EPSG, IGNF, PROJ

      • +

    • metadata: a few key/value pairs, for example to indicate the version of the registries imported in the database

      • +

    • object_view: synthetic view listing objects (ellipsoids, datums, CRS, coordinate operations…) code and name, and the table name where they are further described

      • +

    • alias_names: list possible alias for the name field of object table

      • +

    • link_from_deprecated_to_non_deprecated: to handle the link between old ESRI to new ESRI/EPSG codes

      • +
    +
    +
    +
    • +
  • +
    Common:
      +

    • unit_of_measure: table with UnitOfMeasure definitions.

      • +

    • area: table with area-of-use (bounding boxes) applicable to CRS and coordinate operations.

      • +
    +
    +
    +
    • +
  • +
    Coordinate systems:
      +

    • axis: table with CoordinateSystemAxis definitions.

      • +

    • coordinate_system: table with CoordinateSystem definitions.

      • +
    +
    +
    +
    • +
  • +
    Ellipsoid and datums:
      +

    • ellipsoid: table with ellipsoid definitions.

      • +

    • prime_meridian: table with PrimeMeridian definitions.

      • +

    • geodetic_datum: table with GeodeticReferenceFrame definitions.

      • +

    • vertical_datum: table with VerticalReferenceFrame definitions.

      • +
    +
    +
    +
    • +
  • +
    CRS:
      +

    • geodetic_crs: table with GeodeticCRS and GeographicCRS definitions.

      • +

    • projected_crs: table with ProjectedCRS definitions.

      • +

    • vertical_crs: table with VerticalCRS definitions.

      • +

    • compound_crs: table with CompoundCRS definitions.

      • +
    +
    +
    +
    • +
  • +
    Coordinate operations:
      +

    • coordinate_operation_view: view giving a number of common attributes shared by the concrete tables implementing CoordinateOperation

      • +

    • conversion: table with definitions of Conversion (mostly parameter and values of Projection)

      • +

    • concatenated_operation: table with definitions of ConcatenatedOperation.

      • +

    • grid_transformation: table with all grid-based transformations.

      • +

    • grid_packages: table listing packages in which grids can be found. ie “proj-datumgrid”, “proj-datumgrid-europe”, …

      • +

    • grid_alternatives: table binding official grid names to PROJ grid names. e.g “Und_min2.5x2.5_egm2008_isw=82_WGS84_TideFree.gz” –> “egm08_25.gtx”

      • +

    • helmert_transformation: table with all Helmert-based transformations.

      • +

    • other_transformation: table with other type of transformations.

      • +
    +
    +
    +
    • +
+

The main departure with the structure of the EPSG database is the split of +the various coordinate operations over several tables. This was done mostly +for human-readability as the EPSG organization of coordoperation, +coordoperationmethod, coordoperationparam, coordoperationparamusage, +coordoperationparamvalue tables makes it hard to grasp at once all the +parameters and values for a given operation.

+
+
+
+

Utilities

+

A new projinfo utility has been added. It enables the user to enter a CRS or +coordinate operation by a AUTHORITY:CODE, PROJ string or WKT string, and see +it translated in the different flavors of PROJ and WKT strings. +It also enables to build coordinate operations between two CRSs.

+
+

Usage

+
usage: projinfo [-o formats] [-k crs|operation] [--summary] [-q]
+                [--bbox min_long,min_lat,max_long,max_lat]
+                [--spatial-test contains|intersects]
+                [--crs-extent-use none|both|intersection|smallest]
+                [--grid-check none|discard_missing|sort]
+                [--boundcrs-to-wgs84]
+                {object_definition} | (-s {srs_def} -t {srs_def})
+
+-o: formats is a comma separated combination of: all,default,PROJ4,PROJ,WKT_ALL,WKT2_2015,WKT2_2018,WKT1_GDAL
+    Except 'all' and 'default', other format can be preceded by '-' to disable them
+
+
+
+
+

Examples

+
+

Specify CRS by AUTHORITY:CODE

+
$ projinfo EPSG:4326
+
+PROJ string:
++proj=pipeline +step +proj=longlat +ellps=WGS84 +step +proj=unitconvert +xy_in=rad +xy_out=deg +step +proj=axisswap +order=2,1
+
+WKT2_2015 string:
+GEODCRS["WGS 84",
+    DATUM["World Geodetic System 1984",
+        ELLIPSOID["WGS 84",6378137,298.257223563,
+            LENGTHUNIT["metre",1]]],
+    PRIMEM["Greenwich",0,
+        ANGLEUNIT["degree",0.0174532925199433]],
+    CS[ellipsoidal,2],
+        AXIS["geodetic latitude (Lat)",north,
+            ORDER[1],
+            ANGLEUNIT["degree",0.0174532925199433]],
+        AXIS["geodetic longitude (Lon)",east,
+            ORDER[2],
+            ANGLEUNIT["degree",0.0174532925199433]],
+    AREA["World"],
+    BBOX[-90,-180,90,180],
+    ID["EPSG",4326]]
+
+
+
+
+

Specify CRS by PROJ string and specify output formats

+
$ projinfo -o PROJ4,PROJ,WKT1_GDAL,WKT2_2018 "+title=IGN 1972 Nuku Hiva - UTM fuseau 7 Sud +proj=tmerc +towgs84=165.7320,216.7200,180.5050,-0.6434,-0.4512,-0.0791,7.420400 +a=6378388.0000 +rf=297.0000000000000 +lat_0=0.000000000 +lon_0=-141.000000000 +k_0=0.99960000 +x_0=500000.000 +y_0=10000000.000 +units=m +no_defs"
+
+PROJ string:
+Error when exporting to PROJ string: BoundCRS cannot be exported as a PROJ.5 string, but its baseCRS might
+
+PROJ.4 string:
++proj=utm +zone=7 +south +ellps=intl +towgs84=165.732,216.72,180.505,-0.6434,-0.4512,-0.0791,7.4204
+
+WKT2_2018 string:
+BOUNDCRS[
+    SOURCECRS[
+        PROJCRS["IGN 1972 Nuku Hiva - UTM fuseau 7 Sud",
+            BASEGEOGCRS["unknown",
+                DATUM["unknown",
+                    ELLIPSOID["International 1909 (Hayford)",6378388,297,
+                        LENGTHUNIT["metre",1,
+                            ID["EPSG",9001]]]],
+                PRIMEM["Greenwich",0,
+                    ANGLEUNIT["degree",0.0174532925199433],
+                    ID["EPSG",8901]]],
+            CONVERSION["unknown",
+                METHOD["Transverse Mercator",
+                    ID["EPSG",9807]],
+                PARAMETER["Latitude of natural origin",0,
+                    ANGLEUNIT["degree",0.0174532925199433],
+                    ID["EPSG",8801]],
+                PARAMETER["Longitude of natural origin",-141,
+                    ANGLEUNIT["degree",0.0174532925199433],
+                    ID["EPSG",8802]],
+                PARAMETER["Scale factor at natural origin",0.9996,
+                    SCALEUNIT["unity",1],
+                    ID["EPSG",8805]],
+                PARAMETER["False easting",500000,
+                    LENGTHUNIT["metre",1],
+                    ID["EPSG",8806]],
+                PARAMETER["False northing",10000000,
+                    LENGTHUNIT["metre",1],
+                    ID["EPSG",8807]]],
+            CS[Cartesian,2],
+                AXIS["(E)",east,
+                    ORDER[1],
+                    LENGTHUNIT["metre",1,
+                        ID["EPSG",9001]]],
+                AXIS["(N)",north,
+                    ORDER[2],
+                    LENGTHUNIT["metre",1,
+                        ID["EPSG",9001]]]]],
+    TARGETCRS[
+        GEOGCRS["WGS 84",
+            DATUM["World Geodetic System 1984",
+                ELLIPSOID["WGS 84",6378137,298.257223563,
+                    LENGTHUNIT["metre",1]]],
+            PRIMEM["Greenwich",0,
+                ANGLEUNIT["degree",0.0174532925199433]],
+            CS[ellipsoidal,2],
+                AXIS["latitude",north,
+                    ORDER[1],
+                    ANGLEUNIT["degree",0.0174532925199433]],
+                AXIS["longitude",east,
+                    ORDER[2],
+                    ANGLEUNIT["degree",0.0174532925199433]],
+            ID["EPSG",4326]]],
+    ABRIDGEDTRANSFORMATION["Transformation from unknown to WGS84",
+        METHOD["Position Vector transformation (geog2D domain)",
+            ID["EPSG",9606]],
+        PARAMETER["X-axis translation",165.732,
+            ID["EPSG",8605]],
+        PARAMETER["Y-axis translation",216.72,
+            ID["EPSG",8606]],
+        PARAMETER["Z-axis translation",180.505,
+            ID["EPSG",8607]],
+        PARAMETER["X-axis rotation",-0.6434,
+            ID["EPSG",8608]],
+        PARAMETER["Y-axis rotation",-0.4512,
+            ID["EPSG",8609]],
+        PARAMETER["Z-axis rotation",-0.0791,
+            ID["EPSG",8610]],
+        PARAMETER["Scale difference",1.0000074204,
+            ID["EPSG",8611]]]]
+
+WKT1_GDAL:
+PROJCS["IGN 1972 Nuku Hiva - UTM fuseau 7 Sud",
+    GEOGCS["unknown",
+        DATUM["unknown",
+            SPHEROID["International 1909 (Hayford)",6378388,297],
+            TOWGS84[165.732,216.72,180.505,-0.6434,-0.4512,-0.0791,7.4204]],
+        PRIMEM["Greenwich",0,
+            AUTHORITY["EPSG","8901"]],
+        UNIT["degree",0.0174532925199433,
+            AUTHORITY["EPSG","9122"]],
+        AXIS["Longitude",EAST],
+        AXIS["Latitude",NORTH]],
+    PROJECTION["Transverse_Mercator"],
+    PARAMETER["latitude_of_origin",0],
+    PARAMETER["central_meridian",-141],
+    PARAMETER["scale_factor",0.9996],
+    PARAMETER["false_easting",500000],
+    PARAMETER["false_northing",10000000],
+    UNIT["metre",1,
+        AUTHORITY["EPSG","9001"]],
+    AXIS["Easting",EAST],
+    AXIS["Northing",NORTH]]
+
+
+
+
+

Find transformations between 2 CRS

+

Between “Poland zone I” (based on Pulkovo 42 datum) and “UTM WGS84 zone 34N”

+

Summary view:

+
$ projinfo -s EPSG:2171 -t EPSG:32634 --summary
+
+Candidate operations found: 1
+unknown id, Inverse of Poland zone I + Pulkovo 1942(58) to WGS 84 (1) + UTM zone 34N, 1 m, Poland - onshore
+
+
+

Display of pipelines:

+
$ PROJ_LIB=data src/projinfo -s EPSG:2171 -t EPSG:32634 -o PROJ
+
+PROJ string:
++proj=pipeline +step +proj=axisswap +order=2,1 +step +inv +proj=sterea +lat_0=50.625 +lon_0=21.0833333333333 +k=0.9998 +x_0=4637000 +y_0=5647000 +ellps=krass +step +proj=cart +ellps=krass +step +proj=helmert +x=33.4 +y=-146.6 +z=-76.3 +rx=-0.359 +ry=-0.053 +rz=0.844 +s=-0.84 +convention=position_vector +step +inv +proj=cart +ellps=WGS84 +step +proj=utm +zone=34 +ellps=WGS84
+
+
+
+
+
+
+

Impacted files

+

New files (excluding makefile.am, CMakeLists.txt and other build infrastructure +artefacts):

+
+
+ + + + + +
+
+
+

C API

+

proj.h has been extended to bind a number of C++ classes/methods to a C API.

+

The main structure is an opaque PJ_OBJ* roughly encapsulating a osgeo::proj::BaseObject, +that can represent a CRS or a CoordinateOperation object. A number of the +C functions will work only if the right type of underlying C++ object is used +with them. Misuse will be properly handled at runtime. If a user passes +a PJ_OBJ* representing a coordinate operation to a pj_obj_crs_xxxx() function, +it will properly error out. This design has been chosen over creating a +dedicate PJ_xxx object for each C++ class, because such an approach would +require adding many conversion and free functions for little benefit.

+

This C API is incomplete. In particular, it does not allow to +build ISO19111 objects at hand. However it currently permits a number of +actions:

+
+
    +

  • building CRS and coordinate operations from WKT and PROJ strings, or +from the proj.db database

    • +

  • exporting CRS and coordinate operations as WKT and PROJ strings

    • +

  • querying main attributes of those objects

    • +

  • finding coordinate operations between two CRS.

    • +
+
+

test_c_api.cpp should demonstrates simple usage of the API (note: +for the conveniency of writing the tests in C++, test_c_api.cpp wraps the C PJ_OBJ* +instances in C++ ‘keeper’ objects that automatically call the pj_obj_unref() +function at function end. In a pure C use, the caller must use pj_obj_unref() +to prevent leaks.)

+
+
+

Documentation

+

All public C++ classes and methods and C functions are documented with +Doxygen.

+

Current snapshot of Class list

+

Spaghetti inheritance diagram

+

A basic integration of the Doxygen XML output into the general PROJ +documentation (using reStructuredText format) has been done with the +Sphinx extension Breathe, producing:

+
+
+
+
+

Testing

+

Nearly all exported methods are tested by a unit test. Global line coverage +of the new files is 92%. Those tests represent 16k lines of codes.

+
+
+

Build requirements

+

The new code leverages on a number of C++11 features (auto keyword, constexpr, +initializer list, std::shared_ptr, lambda functions, etc.), which means that +a C++11-compliant compiler must be used to generate PROJ:

+
+
    +

  • gcc >= 4.8

    • +

  • clang >= 3.3

    • +

  • Visual Studio >= 2015.

    • +
+
+

Compilers tested by the Travis-CI and AppVeyor continuous integration +environments:

+
+
    +

  • GCC 4.8

    • +

  • mingw-w64-x86-64 4.8

    • +

  • clang 5.0

    • +

  • Apple LLVM version 9.1.0 (clang-902.0.39.2)

    • +

  • MSVC 2015 32 and 64 bit

    • +

  • MSVC 2017 32 and 64 bit

    • +
+
+

The libsqlite3 >= 3.7 development package must also be available. And the sqlite3 +binary must be available to build the proj.db files from the .sql files.

+
+
+

Runtime requirements

+
    +

  • libc++/libstdc++/MSVC runtime consistent with the compiler used

    • +

  • libsqlite3 >= 3.7

    • +
+
+
+

Backward compatibility

+

At this stage, no backward compatibility issue is foreseen, as no +existing functional C code has been modified to use the new capabilities

+
+
+

Future work

+

The work described in this RFC will be pursued in a number of directions. +Non-exhaustively:

+
+
    +

  • Support for ESRI WKT1 dialect (PROJ currently ingest the ProjectedCRS in +esri.sql in that dialect, but there is no mapping between it and EPSG +operation and parameter names, so conversion to PROJ strings does not +always work.

    • +

  • closer integration with the existing code base. In particular, the +init=dict:code +syntax should now go first to the database (then the epsg and IGNF +files can be removed). Similarly proj_create_crs_to_crs() could use the +new capabilities to find an appropriate coordinate transformation.

    • +

  • and whatever else changes are needed to address GDAL and libgeotiff needs

    • +
+
+
+
+

Adoption status

+

The RFC has been adopted with support from PSC members Kurt Schwehr, Kristian +Evers, Howard Butler and Even Rouault.

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/community/rfc/rfc-3.html b/community/rfc/rfc-3.html new file mode 100644 index 00000000..3719c8f2 --- /dev/null +++ b/community/rfc/rfc-3.html @@ -0,0 +1,291 @@ + + + + + + + PROJ RFC 3: Dependency management — PROJ 9.0.0 documentation + + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

PROJ RFC 3: Dependency management

+
+
Author
+

Kristian Evers

+
+
Contact
+

kreve@sdfe.dk

+
+
Status
+

Adopted

+
+
Last Updated
+

2019-01-16

+
+
+
+

Summary

+

This document defines a set of guidelines for dependency management in PROJ. +With PROJ being a core component in many downstream software packages clearly +stating which dependencies the library has is of great value. This document +concern both programming language standards as well as minimum required +versions of library dependencies and build tools.

+

It is proposed to adopt a rolling update scheme that ensures that PROJ is +sufficiently accessible, even on older systems, as well as keeping up with the +technological evolution. The scheme is divided in two parts, one concerning +versions of used programming languages within PROJ and the other concerning +software packages that PROJ depend on.

+

With adoption of this RFC, versions used for

+
    +

  1. programming languages will always be at least two revisions behind the most +recent standard

    2. +

  2. software packages will always be at least two years old +(patch releases are exempt)

    3. +
+

A change in programming language standard can only be introduced with a new +major version release of PROJ. Changes for software package dependencies can be +introduced with minor version releases of PROJ. Changing the version +requirements for a dependency needs to be approved by the PSC.

+

Following the above rule set will ensure that all but the most conservative +users of PROJ will be able to build and use the most recent version of the +library.

+

In the sections below details concerning programming languages and software +dependencies are outlined. The RFC is concluded with a bootstrapping section +that details the state of dependencies after the accept of the RFC.

+
+
+

Background

+

PROJ has traditionally been written in C89. Until recently, no formal +requirements of e.g. build systems has been defined and formally accepted by +the project. RFC2 formally introduces dependencies on C++11 and +SQLite 3.7.

+

In this RFC a rolling update of version or standard requirements is described. +The reasoning behind a rolling update scheme is that it has become increasingly +evident that C89 is becoming outdated and creating a less than optimal +development environment for contributors. It has been noted that the most +commonly used compilers all now support more recent versions of C, so the +strict usage of C89 is no longer as critical as it used to be.

+

Similarly, rolling updates to other tools and libraries that PROJ depend on +will ensure that the code base can be kept modern and in line with the rest of +the open source software ecosphere.

+
+
+

C and C++

+

Following RFC2 PROJ is written in both C and C++. At the time of +writing the core library is C based and the code described in RFC2 is written +in C++. While the core library is mostly written in C it is compiled as C++. +Minor sections of PROJ, like the geodesic algorithms are still compiled as C +since there is no apparent benefit of compiling with a C++ compiler. This may +change in the future.

+

Both the C and C++ standards are updated with regular intervals. After an +update of a standard it takes time for compiler manufacturers to implement the +standards fully, which makes adaption of new standards potentially troublesome +if done too soon. On the other hand, waiting too long to adopt new standards +will eventually make the code base feel old and new contributors are more +likely to stay away because they don’t want to work using tools of the past. +With a rolling update scheme both concerns can be managed by always staying +behind the most recent standard, but not so far away that potential +contributors are scared away. Keeping a policy of always lagging behind be two +iterations of the standard is thought to be the best comprise between the two +concerns.

+

C comes in four ISO standardised varieties: C89, C99, C11, C18. In this +document we refer to their informal names for ease of reading. C++ exists in +five varieties: C++98, C++03, C++11, C++14, C++17. Before adoption of this RFC +PROJ uses C89 and C++11. For C, that means that the used standard is three +iterations behind the most recent standard. C++ is two iterations behind. +Following the rules in this RFC the required C standard used in PROJ is at +allowed to be two iterations behind the most recent standard. That means that a +change to C99 is possible, as long as the PROJ PSC acknowledges such a change.

+

When a new standard for either C or C++ is released PROJ should consider +changing its requirement to the next standard in the line. For C++ that means a +change in standard roughly every three years, for C the periods between +standard updates is expected to be longer. Adaptation of new programming +language standards should be coordinated with a major version release of PROJ.

+
+
+

Software dependencies

+

At the time of writing PROJ is dependent on very few external packages. In +fact only one runtime dependency is present: SQLite. Building PROJ also +requires one of two external dependencies for configuration: Autotools or +CMake.

+

As with programming language standards it is preferable that software +dependencies are a bit behind the most recent development. For this reason it +is required that the minimum version supported in PROJ dependencies is at least +two years old, preferably more. It is not a requirement that the minimum +version of dependencies is always kept strictly two years behind current +development, but it is allowed in case future development of PROJ warrants an +update. Changes in minimum version requirements are allowed to happen with +minor version releases of PROJ.

+

At the time of writing the minimum version required for SQLite it 3.7 which was +released in 2010. CMake currently is required to be at least at version 2.8.3 +which was also released in 2010.

+
+
+

Bootstrapping

+

This RFC comes with a set of guidelines for handling dependencies for PROJ in +the future. Up until now dependencies hasn’t been handled consistently, with +some dependencies not being approved formally by the projects governing body. +Therefore minimum versions of PROJ dependencies is proposed so that at the +acception of this RFC PROJ will have the following external requirements:

+
    +

  • C99 (was C89)

    • +

  • C++11 (already approved in RFC2)

    • +

  • SQLite 3.7 (already approved in RFC2)

    • +

  • CMake 3.5 (was 2.8.3)

    • +
+
+
+

Adoption status

+

The RFC was adopted on 2018-01-19 with +1’s from the following PSC members

+
    +

  • Kristian Evers

    • +

  • Even Rouault

    • +

  • Thomas Knudsen

    • +

  • Howard Butler

    • +
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/community/rfc/rfc-4.html b/community/rfc/rfc-4.html new file mode 100644 index 00000000..03b40b69 --- /dev/null +++ b/community/rfc/rfc-4.html @@ -0,0 +1,881 @@ + + + + + + + PROJ RFC 4: Remote access to grids and GeoTIFF grids — PROJ 9.0.0 documentation + + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

PROJ RFC 4: Remote access to grids and GeoTIFF grids

+
+
Author
+

Even Rouault, Howard Butler

+
+
Contact
+

even.rouault@spatialys.com, howard@hobu.co

+
+
Status
+

Adopted

+
+
Implementation target
+

PROJ 7

+
+
Last Updated
+

2020-01-10

+
+
+
+

Motivation

+

PROJ 6 brings undeniable advances in the management of coordinate +transformations between datums by relying and applying information available in +the PROJ database. PROJ’s rapid evolution from a cartographic projections +library with a little bit of geodetic capability to a full geodetic +transformation and description environment has highlighted the importance of +the support data. Users desire the convenience of software doing the right +thing with the least amount of fuss, and survey organizations wish to deliver +their models across as wide a software footprint as possible. To get results +with the highest precision, a grid file that defines a model that provides +dimension shifts is often needed. The proj-datumgrid project centralizes grids +available under an open data license and bundles them in different archives +split along major geographical regions of the world .

+

It is assumed that a PROJ user has downloaded and installed grid files that are +referred to in the PROJ database. These files can be quite large in aggregate, +and packaging support by major distribution channels is somewhat uneven due to +their size, sometimes ambiguous licensing story, and difficult-to-track +versioning and lineage. It is not always clear to the user, especially to +those who may not be so familiar with geodetic operations, that the highest +precision transformation may not always being applied if grid data is not +available. Users want both convenience and correctness, and management of the +shift files can be challenging to those who may not be aware of their +importance to the process.

+

The computing environment in which PROJ operates is also changing. Because the +shift data can be so large (currently more than 700 MB of uncompressed data, +and growing), deployment of high accuracy operations can be limited due to +deployment size constraints (serverless operations, for example). Changing to a +delivery format that supports incremental access over a network along with +convenient access and compression will ease the resource burden the shift files +present while allowing the project to deliver transformation capability with +the highest known precision provided by the survey organizations.

+

Adjustment grids also tend to be provided in many different formats depending +on the organization and country that produced them. In PROJ, we have over time +“standardized” on using horizontal shift grids as NTv2 and vertical shift grids +using GTX. Both have poor general support as dedicated formats, limited +metadata capabilities, and neither are not necessarily “cloud optimized” for +incremental access across a network.

+
+
+

Summary of work planned by this RFC

+
    +

  • Grids will be hosted by one or several Content Delivery Networks (CDN)

    • +

  • Grid loading mechanism will be reworked to be able to download grids or parts +of grids from a online repository. When opted in, users will no longer have to +manually fetch grid files and place them in PROJ_LIB. +Full and accurate capability of the software will no longer require hundreds +of megabytes of grid shift files in advance, even if only just a few of them +are needed for the transformations done by the user.

    • +

  • Local caching of grid files, or even part of files, so that users end up +mirroring what they actually use.

    • +

  • A grid shift format, for both horizontal and vertical shift grids (and in +potential future steps, for other needs, such as deformation models) will be +implemented.

    • +
+

The use of grids locally available will of course still be available, and will +be the default behavior.

+
+
+

Network access to grids

+

curl will be an optional build dependency of PROJ, added in autoconf and cmake +build systems. It can be disabled at build time, but this must be +an explicit setting of configure/cmake as the resulting builds have less functionality. +When curl is enabled at build time, download of grids themselves will not be +enabled by default at runtime. It will require explicit consent of the user, either +through the API +(proj_context_set_enable_network()) through the PROJ_NETWORK=ON +environment variable, or the network = on setting of proj.ini.

+

Regarding the minimum version of libcurl required, given GDAL experience that +can build with rather ancient libcurl for similar functionality, we can aim for +libcurl >= 7.29.0 (as being available in RHEL 7).

+

An alternate pluggable network interface can also be set by the user in case +support for libcurl was not built in, or if for the desired context of use, the +user wishes to provide the network implementation (a typical use case could be +QGIS that would use its QT-based networking facilities to solve issues with +SSL, proxy, authentication, etc.)

+

A text configuration file, installed in ${installation_prefix}/share/proj/proj.ini +(or ${PROJ_LIB}/proj.ini) +will contain the URL of the CDN that will be used. +The user may also override this setting with the +proj_context_set_url_endpoint() or through the PROJ_NETWORK_ENDPOINT +environment variable.

+

The rationale for putting proj.ini in that location is +that it is a well-known place by PROJ users, with the existing PROJ_LIB mechanics +for systems like Windows where hardcoded paths at runtime aren’t generally usable.

+
+

C API

+

The preliminary C API for the above is:

+
/** Enable or disable network access.
+*
+* @param ctx PROJ context, or NULL
+* @return TRUE if network access is possible. That is either libcurl is
+*         available, or an alternate interface has been set.
+*/
+int proj_context_set_enable_network(PJ_CONTEXT* ctx, int enable);
+
+/** Define URL endpoint to query for remote grids.
+*
+* This overrides the default endpoint in the PROJ configuration file or with
+* the PROJ_NETWORK_ENDPOINT environment variable.
+*
+* @param ctx PROJ context, or NULL
+* @param url Endpoint URL. Must NOT be NULL.
+*/
+void proj_context_set_url_endpoint(PJ_CONTEXT* ctx, const char* url);
+
+/** Opaque structure for PROJ. Implementations might cast it to their
+ * structure/class of choice. */
+typedef struct PROJ_NETWORK_HANDLE PROJ_NETWORK_HANDLE;
+
+/** Network access: open callback
+*
+* Should try to read the size_to_read first bytes at the specified offset of
+* the file given by URL url,
+* and write them to buffer. *out_size_read should be updated with the actual
+* amount of bytes read (== size_to_read if the file is larger than size_to_read).
+* During this read, the implementation should make sure to store the HTTP
+* headers from the server response to be able to respond to
+* proj_network_get_header_value_cbk_type callback.
+*
+* error_string_max_size should be the maximum size that can be written into
+* the out_error_string buffer (including terminating nul character).
+*
+* @return a non-NULL opaque handle in case of success.
+*/
+typedef PROJ_NETWORK_HANDLE* (*proj_network_open_cbk_type)(
+                                                    PJ_CONTEXT* ctx,
+                                                    const char* url,
+                                                    unsigned long long offset,
+                                                    size_t size_to_read,
+                                                    void* buffer,
+                                                    size_t* out_size_read,
+                                                    size_t error_string_max_size,
+                                                    char* out_error_string,
+                                                    void* user_data);
+
+/** Network access: close callback */
+typedef void (*proj_network_close_cbk_type)(PJ_CONTEXT* ctx,
+                                            PROJ_NETWORK_HANDLE* handle,
+                                            void* user_data);
+
+/** Network access: get HTTP headers */
+typedef const char* (*proj_network_get_header_value_cbk_type)(
+                                            PJ_CONTEXT* ctx,
+                                            PROJ_NETWORK_HANDLE* handle,
+                                            const char* header_name,
+                                            void* user_data);
+
+/** Network access: read range
+*
+* Read size_to_read bytes from handle, starting at offset, into
+* buffer.
+* During this read, the implementation should make sure to store the HTTP
+* headers from the server response to be able to respond to
+* proj_network_get_header_value_cbk_type callback.
+*
+* error_string_max_size should be the maximum size that can be written into
+* the out_error_string buffer (including terminating nul character).
+*
+* @return the number of bytes actually read (0 in case of error)
+*/
+typedef size_t (*proj_network_read_range_type)(
+                                            PJ_CONTEXT* ctx,
+                                            PROJ_NETWORK_HANDLE* handle,
+                                            unsigned long long offset,
+                                            size_t size_to_read,
+                                            void* buffer,
+                                            size_t error_string_max_size,
+                                            char* out_error_string,
+                                            void* user_data);
+
+/** Define a custom set of callbacks for network access.
+*
+* All callbacks should be provided (non NULL pointers).
+*
+* @param ctx PROJ context, or NULL
+* @param open_cbk Callback to open a remote file given its URL
+* @param close_cbk Callback to close a remote file.
+* @param get_header_value_cbk Callback to get HTTP headers
+* @param read_range_cbk Callback to read a range of bytes inside a remote file.
+* @param user_data Arbitrary pointer provided by the user, and passed to the
+* above callbacks. May be NULL.
+* @return TRUE in case of success.
+*/
+int proj_context_set_network_callbacks(
+    PJ_CONTEXT* ctx,
+    proj_network_open_cbk_type open_cbk,
+    proj_network_close_cbk_type close_cbk,
+    proj_network_get_header_value_cbk_type get_header_value_cbk,
+    proj_network_read_range_type read_range_cbk,
+    void* user_data);
+
+
+

To make network access efficient, PROJ will internally have a in-memory cache +of file ranges to only issue network requests by chunks of 16 KB or multiple of them, +to limit the number of HTTP GET requests and minimize latency caused by network +access. This is very similar to the behavior of the GDAL +/vsicurl/ +I/O layer. The plan is to mostly copy GDAL’s vsicurl implementation inside PROJ, with +needed adjustments and proper namespacing of it.

+

A retry strategy (typically a delay with an exponential back-off and some random +jitter) will be added to account for intermittent network or server-side failure.

+
+
+

URL building

+

The PROJ database has a grid_transformation grid whose column grid_name +(and possibly grid2_name) contain the name of the grid as indicated by the +authority having registered the transformation (typically EPSG). As those +grid names are not generally directly usable by PROJ, the PROJ database has +also a grid_alternatives table that link original grid names to the ones used +by PROJ. When network access will be available and needed due to lack of a +local grid, the full URL will be the +endpoint from the configuration or set by the user, the basename of the PROJ +usable filename, and the “tif” suffix. So if the CDN is at http://example.com +and the name from grid_alternatives is egm96_15.gtx, then the URL will +be http://example.com/egm96_15.tif

+
+
+

Grid loading

+

The following files will be affected, in one way or another, by the above describes +changes: +nad_cvt.cpp, nad_intr.cpp, nad_init.cpp, grid_info.cpp, grid_list.cpp, apply_gridshift.cpp, +apply_vgridshift.cpp.

+

In particular the current logic that consists to ingest all the values of a +grid/subgrid in the ct->cvs array will be completely modified, to enable +access to grid values at a specified (x,y) location.

+
+
+

proj_create_crs_to_crs() / proj_create_operations() impacts

+

Once network access is available, all grids known to the PROJ database +(grid_transformation + grid_alternatives table) will be assumed to be available, +when computing the potential pipelines between two CRS.

+

Concretely, this will be equivalent to calling +proj_operation_factory_context_set_grid_availability_use() +with the use argument set to a new enumeration value

+
/** Results will be presented as if grids known to PROJ (that is
+* registered in the grid_alternatives table of its database) were
+* available. Used typically when networking is enabled.
+*/
+PROJ_GRID_AVAILABILITY_KNOWN_AVAILABLE
+
+
+
+
+

Local on-disk caching of remote grids

+

As many workflows will tend to use the same grids over and over, a local +on-disk caching of remote grids will be added. The cache will be a single +SQLite3 database, in a user-writable directory shared by all applications using +PROJ.

+

Its total size will be configurable, with a default maximum size of 100 MB +in proj.ini. The cache will also keep the timestamp of the last time it checked +various global properties of the file (its size, Last-Modified and ETag headers). +A time-to-live parameter, with a default of 1 day in proj.ini, will be used to +determine whether the CDN should be hit to verify if the information in the +cache is still up-to-date.

+
/** Enable or disable the local cache of grid chunks
+*
+* This overrides the setting in the PROJ configuration file.
+*
+* @param ctx PROJ context, or NULL
+* @param enabled TRUE if the cache is enabled.
+*/
+void proj_grid_cache_set_enable(PJ_CONTEXT *ctx, int enabled);
+
+/** Override, for the considered context, the path and file of the local
+* cache of grid chunks.
+*
+* @param ctx PROJ context, or NULL
+* @param fullname Full name to the cache (encoded in UTF-8). If set to NULL,
+*                 caching will be disabled.
+*/
+void proj_grid_cache_set_filename(PJ_CONTEXT* ctx, const char* fullname);
+
+/** Override, for the considered context, the maximum size of the local
+* cache of grid chunks.
+*
+* @param ctx PROJ context, or NULL
+* @param max_size_MB Maximum size, in mega-bytes (1024*1024 bytes), or
+*                    negative value to set unlimited size.
+*/
+void proj_grid_cache_set_max_size(PJ_CONTEXT* ctx, int max_size_MB);
+
+/** Override, for the considered context, the time-to-live delay for
+* re-checking if the cached properties of files are still up-to-date.
+*
+* @param ctx PROJ context, or NULL
+* @param ttl_seconds Delay in seconds. Use negative value for no expiration.
+*/
+void proj_grid_cache_set_ttl(PJ_CONTEXT* ctx, int ttl_seconds);
+
+/** Clear the local cache of grid chunks.
+ *
+ * @param ctx PROJ context, or NULL.
+ */
+void proj_grid_cache_clear(PJ_CONTEXT* ctx);
+
+
+

The planned database structure is:

+
-- General properties on a file
+CREATE TABLE properties(
+ url          TEXT PRIMARY KEY NOT NULL,
+ lastChecked  TIMESTAMP NOT NULL,
+ fileSize     INTEGER NOT NULL,
+ lastModified TEXT,
+ etag         TEXT
+);
+
+-- Store chunks of data. To avoid any potential fragmentation of the
+-- cache, the data BLOB is always set to the maximum chunk size of 16 KB
+-- (right padded with 0-byte)
+-- The actual size is stored in chunks.data_size
+CREATE TABLE chunk_data(
+ id        INTEGER PRIMARY KEY AUTOINCREMENT CHECK (id > 0),
+ data      BLOB NOT NULL
+);
+
+-- Record chunks of data by (url, offset)
+CREATE TABLE chunks(
+ id        INTEGER PRIMARY KEY AUTOINCREMENT CHECK (id > 0),
+ url       TEXT NOT NULL,
+ offset    INTEGER NOT NULL,
+ data_id   INTEGER NOT NULL,
+ data_size INTEGER NOT NULL,
+ CONSTRAINT fk_chunks_url FOREIGN KEY (url) REFERENCES properties(url),
+ CONSTRAINT fk_chunks_data FOREIGN KEY (data_id) REFERENCES chunk_data(id)
+);
+CREATE INDEX idx_chunks ON chunks(url, offset);
+
+-- Doubly linked list of chunks. The next link is to go to the least-recently
+-- used entries.
+CREATE TABLE linked_chunks(
+ id        INTEGER PRIMARY KEY AUTOINCREMENT CHECK (id > 0),
+ chunk_id  INTEGER NOT NULL,
+ prev      INTEGER,
+ next      INTEGER,
+ CONSTRAINT fk_links_chunkid FOREIGN KEY (chunk_id) REFERENCES chunks(id),
+ CONSTRAINT fk_links_prev FOREIGN KEY (prev) REFERENCES linked_chunks(id),
+ CONSTRAINT fk_links_next FOREIGN KEY (next) REFERENCES linked_chunks(id)
+);
+CREATE INDEX idx_linked_chunks_chunk_id ON linked_chunks(chunk_id);
+
+-- Head and tail pointers of the linked_chunks. The head pointer is for
+-- the most-recently used chunk.
+-- There should be just one row in this table.
+CREATE TABLE linked_chunks_head_tail(
+  head       INTEGER,
+  tail       INTEGER,
+  CONSTRAINT lht_head FOREIGN KEY (head) REFERENCES linked_chunks(id),
+  CONSTRAINT lht_tail FOREIGN KEY (tail) REFERENCES linked_chunks(id)
+);
+INSERT INTO linked_chunks_head_tail VALUES (NULL, NULL);
+
+
+

The chunks table will store 16 KB chunks (or less for terminating chunks). +The linked_chunks and linked_chunks_head_tail table swill act as a doubly linked +list of chunks, with the least recently used ones at the end of the list, which +will be evicted when the cache saturates.

+

The directory used to locate this database will be ${XDG_DATA_HOME}/proj +(per https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) +where ${XDG_DATA_HOME} defaults to ${HOME}/.local/share on Unix builds +and ${LOCALAPPDATA} on Windows builds. Exact details to be sorted out, but +https://github.com/ActiveState/appdirs/blob/a54ea98feed0a7593475b94de3a359e9e1fe8fdb/appdirs.py#L45-L97 +can be a good reference.

+

As this database might be accessed by several threads or processes at the same +time, the code accessing to it will carefully honour SQLite3 errors regarding +to locks, to do appropriate retries if another thread/process is currently +locking the database. Accesses requiring a modification of the database will +start with a BEGIN IMMEDIATE transaction so as to acquire a write lock.

+
+

Note

+

This database should be hosted on a local disk, not a network one. +Otherwise SQLite3 locking issues are to be expected.

+
+
+
+

CDN provider

+

Amazon Public Datasets +has offered to be a storage and CDN provider.

+

The program covers storage and egress (bandwidth) of the data. +They generally don’t allow usage of CloudFront +(their CDN) as part of the program (we would usually look to have it covered +by credits), but in this instance, they would be fine to provide it. +They’d only ask that we keep the CloudFront URL “visible” (as appropriate for +the use case) so people can see where the data is hosted in case they go looking. +Their terms can be seen at https://aws.amazon.com/service-terms/ and CloudFront +has its own, small section. Those terms may change a bit from time to time for +minor changes. Major changing service terms is assumed to be unfrequent. +There are also the Public Dataset Program terms at http://aws.amazon.com/public-datasets/terms/. +Those also do not effectively change over time and are renewed on a 2 year basis.

+
+
+

Criteria for grid hosting

+

The grids hosted on the CDN will be exactly the ones collected, +currently and in the future, by the proj-datumgrid +initiative. In particular, new grids are accepted as long as +they are released under a license that is compatible with the +Open Source Definition and the source +of the grid is clearly stated and verifiable. Suitable licenses include:

+
    +

  • Public domain

    • +

  • X/MIT

    • +

  • BSD 2/3/4 clause

    • +

  • CC0

    • +

  • CC-BY (v3.0 or later)

    • +

  • CC-BY-SA (v3.0 or later)

    • +
+

For new grids to be transparently used by the proj_create_crs_to_crs() mechanics, +they must be registered in the PROJ database (proj.db) in the grid_transformation and +grid_alternatives table. The nominal path to have a new record in the grid_transformation +is to have a transformation being registered in the EPSG dataset (if there is no +existing one), which will be subsequently imported into the PROJ database.

+
+
+

Versioning, historical preservation of grids

+

The policy regarding this should be similar to the one applied to +proj-datumgrid, which even if +not formalized, is around the following lines:

+
    +

  • Geodetic agencies release regularly new version of grids. Typically for the +USA, NOAA has released GEOID99, GEOID03, GEOID06, GEOID09, GEOID12A, GEOID12B, +GEOID18 for the NAVD88 to NAD83/NAD83(2011) vertical adjustments. Each of these +grids is considered by EPSG and PROJ has a separate object, with distinct filenames. +The release of a new version does not cause the old grid to be automatically removed. +That said, due to advertized accuracies and supersession rules of the EPSG dataset, the +most recent grid will generally be used for a CRS -> CRS transformation if the +user uses proj_create_crs_to_crs() (with the exception that if a VERT_CRS WKT +includes a GEOID_MODEL known to PROJ, an old version of the grid will be used). +If the user specifies a whole pipeline with an explicit grid name, it will be +of course strictly honoured. +As time goes, the size of the datasets managed by proj-datumgrid will be increasing, +we will have to explore on we managed that for the distributed .zip / .tar.gz +archives. This should not be a concern for CDN hosted content.

    • +

  • In case software-related conversion errors from the original grid format to the +one used by PROJ (be it GTX, NTv2 or GeoTIFF) would happen, the previous erroneous +version of the dataset would be replaced by the corrected one. In that situation, +this might have an effect with the local on-disk caching of remote grids. We will +have to see with the CDN providers used if we can use for example the ETag HTTP header +on the client to detect a change, so that old cached content is not erroneously +reused (if not possible, we’ll have to use some text file listing the grid names and their +current md5sum)

    • +
+
+
+
+

Grids in GeoTIFF format

+
+

Limitations of current formats

+

Several formats exist depending on the ad-hoc needs and ideas of the original +data producer. It would be appropriate to converge on a common format able to +address the different use cases.

+
    +

  • Not tiled. Tiling is a nice to have property for cloud-friendly access to +large files.

    • +

  • No support for compression

    • +

  • The NTv2 structures is roughly: header of main grid, data of main grid, +header of subgrid 1, data of subgrid 1, header of subgrid 2, data of subgrid 2, +etc.Due to the headers being scattered through the file, it is not possibly +to retrieve with a single HTTP GET request all header information.

    • +

  • GTX format has no provision to store metadata besides the minimum georeferencing +of the grid. NTv2 is a bit richer, but no extensible metadata possible.

    • +
+
+
+

Discussion on choice of format

+

We have been made recently aware of other initiatives from the industry to come +with a common format to store geodetic adjustment data. Some discussions have +happen recently within the OGC CRS Working group. Past efforts include the +Esri’s proposed Geodetic data Grid eXchange Format, GGXF, briefly mentioned at +page 86 of +https://iag.dgfi.tum.de/fileadmin/IAG-docs/Travaux2015/01_Travaux_Template_Comm_1_tvd.pdf +and page 66 of ftp://ftp.iaspei.org/pub/meetings/2010-2019/2015-Prague/IAG-Geodesy.pdf +The current trend of those works would be to use a netCDF / HDF5 container.

+

So, for the sake of completeness, we list hereafter a few potential candidate +formats and their pros and cons.

+
+

TIFF/GeoTIFF

+

Strong points:

+
    +

  • TIFF is a well-known and widespread format.

    • +

  • The GeoTIFF encoding is a widely industry supported scheme to encode georeferencing. +It is now a OGC standard

    • +

  • There are independent initiatives to share grids as GeoTIFF, like +that one

    • +

  • TIFF can contain multiple images (IFD: Image File Directory) chained together. +This is the mechanism used for multiple-page scanned TIFF files, or in the +geospatial field to store multi-resolution/pyramid rasters. So it can be +used with sub-grids as in the NTv2 format.

    • +

  • Extensive experience with the TIFF format, and its appropriateness for network +access, in particular through the Cloud Optimized GeoTIFF initiative +whose layout can make use of sub-grids efficient from a network access +perspective, because grid headers can be put at the beginning of the file, and +so being retrieved in a single HTTP GET request.

    • +

  • TIFF can be tiled.

    • +

  • TIFF can be compressed. Commonly found compression formats are DEFLATE, LZW, +combined with differential integer or floating point predictors

    • +

  • A TIFF image can contain a configurable number of channels/bands/samples. +In the rest of the document, we will use the sample terminology for this concept.

    • +

  • TIFF sample organization can be configured: either the values of different +samples are packed together (PlanarConfiguration = Contig), or put in separate tiles/strips +(PlanarConfiguration = Separate)

    • +

  • libtiff is a dependency commonly found in binary distributions of the +“ecosystem” to which PROJ belongs too

    • +

  • libtiff benefits from many years of efforts to increase its security, for +example being integrated to the oss-fuzz initiative. Given the potential +fetching of grids, using security tested components is an important concern.

    • +

  • Browser-side: there are “ports” of libtiff/libgeotiff in the browser such +as https://geotiffjs.github.io/ which could potentially make a port of PROJ +easier.

    • +
+

Weak points:

+
    +

  • we cannot use libgeotiff, since it depends itself on PROJ (to resolve CRS +or components of CRS from their EPSG codes). That said, for PROJ intended +use, we only need to decode the ModelTiepointTag and ModelPixelScaleTag TIFF +tags, so this can be done “at hand”

    • +

  • the metadata capabilities of TIFF baseline are limited. The TIFF format comes +with a predefined set of metadata items whose keys have numeric values. That +said, GDAL has used for the last 20 years or so a dedicated tag, +GDAL_METADATA +of code 42112 that holds a XML-formatted string being able to store arbitrary +key-pair values.

    • +
+
+
+

netCDF v3

+

Strong points:

+
    +

  • The binary format description as given in +OGC 10-092r3 is relatively simple, +but it would still probably be necessary to use libnetcdf-c to access it

    • +

  • Metadata can be stored easily in netCDF attributes

    • +
+

Weak points:

+
    +

  • No compression in netCDF v3

    • +

  • No tiling in netCDF v3

    • +

  • Multi-samples variables are located in different sections of the files +(correspond to TIFF PlanarConfiguration = Separate)

    • +

  • No natural way of having hierarchical / multigrids. They must be encoded as +separate variables

    • +

  • georeferencing in netCDF is somewhat less standardized than TIFF/GeoTIFF. +The generally used model is the conventions for CF (Climate and Forecast) +metadata +but there is nothing really handy in them for simple georeferencing with +the coordinate of the upper-left pixel and the resolution. The practice is +to write explicit lon and lat variables with all values taken by the grid. +GDAL has for many years supported a simpler syntax, using a GeoTransform +attribute.

    • +

  • From the format description, its layout could be relatively cloud friendly, +except that libnetcdf has no API to plug an alternate I/O layer.

    • +

  • Most binary distributions of netCDF nowadays are based on libnetcdf v4, which +implies the HDF5 dependency.

    • +

  • From a few issues we identified a few years ago regarding crashes on corrupted +datasets, we contacted libnetcdf upstream, but they did not seem to be +interested in addressing those security issues.

    • +
+
+
+

netCDF v4 / HDF5

+

Note: The netCDF v4 format is a profile of the HDF5 file format.

+

Strong points:

+
    +

  • Compression supported (ZLIB and SZIP predefined)

    • +

  • Tiling (chunking) supported

    • +

  • Values of Multi-sample variables can be interleaved together (similarly +to TIFF PlanarConfiguration = Contig) by using compound data types.

    • +

  • Hierarchical organization with groups

    • +

  • While the netCDF API does not provide an alternate I/O layer, this is +possible with the HDF5 API.

    • +

  • Grids can be indexed by more than 2 dimensions (for current needs, we +don’t need more than 2D support)

    • +
+

Weak points:

+
    +

  • The HDF 5 File format +is more complex than netCDF v3, and likely more than TIFF. We do not have +in-depth expertise of it to assess its cloud-friendliness.

    • +

  • The ones mentioned for netCDF v3 regarding georeferencing and security apply.

    • +
+
+
+

GeoPackage

+

As PROJ has already a SQLite3 dependency, GeoPackage could be examined as a +potential solution.

+

Strong points:

+
    +

  • SQLite3 dependency

    • +

  • OGC standard

    • +

  • Multi-grid capabilities

    • +

  • Tiling

    • +

  • Compression

    • +

  • Metadata capabilities

    • +
+

Weak points:

+
    +

  • GeoPackage mostly address the RGB(A) Byte use case, or via the tile gridded +data extension, single-sample non-Byte data. No native support for multi-sample +non-Byte data: each sample should be put in a separate raster table.

    • +

  • Experience shows that SQLite3 layout (at least the layout adopted when using +the standard libsqlite3) is not cloud friendly. Indices may be scattered in +different places of the file.

    • +
+
+
+

Conclusions

+

The 2 major contenders regarding our goals and constraints are GeoTIFF and HDF5. +Given past positive experience and its long history, GeoTIFF remains our preferred +choice.

+
+
+
+

Format description

+

The format description is available in a dedicated Geodetic TIFF grids (GTG) +document.

+
+
+

Tooling

+

A script will be developed to accept a list of individual grids to combine +together into a single file.

+

A ntv2_to_gtiff.py convenience script will be created to convert NTv2 grids, +including their subgrids, to the above +described GeoTIFF layout.

+

A validation Python script will be created to check that a file meets the above +described requirements and recommendations.

+
+
+

Build requirements

+

The minimum libtiff version will be 4.0 (RHEL 7 ships with libtiff 4.0.3). +To be able to read grids stored on the CDN, libtiff will need to build against +zlib to have DEFLATE and LZW support, which is met by all known binary distributions +of libtiff.

+

The libtiff dependency can be disabled at build time, but this must be +an explicit setting of configure/cmake as the resulting builds have less functionality.

+
+
+
+

Dropping grid catalog functionality

+

While digging through existing code, I more or less discovered that the PROJ +code base has the concept of a grid catalog. This is a feature apparently triggered by +using the +catalog=somefilename.csv in a PROJ string, where the CSV file list +grid names, their extent, priority and date. It seems to be an alternative to using ++nadgrids with multiple grids, with the extra ability to interpolate shift values between +several grids if a +date parameter is provided and the grid catalog mentions a +date for each grids. +It was added in June 2012 per commit fcb186942ec8532655ff6cf4cc990e5da669a3bc

+

This feature is likely unknown to most users as there is no known documentation for +it (neither in current documentation, nor in historic one). +It is not either tested by PROJ tests, so its working status is unknown. It would +likely make implementation of this RFC easier if this was removed. This would result in +completely dropping the gridcatalog.cpp and gc_reader.cpp files, their call sites +and the catalog_name and datum_date parameter from the PJ structure.

+

In case similar functionality would be be needed, it might be later reintroduced +as an extra mode of Horizontal grid shift, or using a dedicated transformation method, +similarly to the Kinematic datum shifting utilizing a deformation model one, +and possibly combining the several grids to interpolate among in the same file, +with a date metadata item.

+
+
+

Backward compatibility issues

+

None anticipated, except the removal of the (presumably little used) grid catalog +functionality.

+
+ +
+

Documentation

+
    +

  • New API function will be documented.

    • +

  • A dedicated documentation page will be created to explain the working of +network-based access.

    • +

  • A dedicated documentation page will be created to describe the GeoTIFF based +grid format. Mostly reusing above material.

    • +
+
+
+

Testing

+

Number of GeoTIFF formulations (tiled vs untiled, PlanarConfiguration Separate vs +Contig, data types, scale+offset vs not, etc.) will be tested.

+

For testing of network capabilities, a mix of real hits to the CDN and use of +the alternate pluggable network interface to test edge cases will be used.

+
+
+

Proposed implementation

+

A proposed implementation is available at https://github.com/OSGeo/PROJ/pull/1817

+

Tooling scripts are currently available at https://github.com/rouault/sample_proj_gtiff_grids/ +(will be ultimately stored in PROJ repository)

+
+
+

Adoption status

+

The RFC was adopted on 2020-01-10 with +1’s from the following PSC members

+
    +

  • Kristian Evers

    • +

  • Even Rouault

    • +

  • Thomas Knudsen

    • +

  • Howard Butler

    • +

  • Kurt Schwehr

    • +
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/community/rfc/rfc-5.html b/community/rfc/rfc-5.html new file mode 100644 index 00000000..c709c5ed --- /dev/null +++ b/community/rfc/rfc-5.html @@ -0,0 +1,280 @@ + + + + + + + PROJ RFC 5: Adopt GeoTIFF-based grids for grids delivered with PROJ — PROJ 9.0.0 documentation + + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

PROJ RFC 5: Adopt GeoTIFF-based grids for grids delivered with PROJ

+
+
Author
+

Even Rouault

+
+
Contact
+

even.rouault@spatialys.com

+
+
Status
+

Adopted

+
+
Implementation target
+

PROJ 7

+
+
Last Updated
+

2020-01-28

+
+
+
+

Motivation

+

This RFC is a continuation of PROJ RFC 4: Remote access to grids and GeoTIFF grids. With RFC4, PROJ can, upon request +of the user, download grids from a CDN in a progressive way. There is also API, +such as proj_download_file() to be able to download a GeoTIFF grid in +the user writable directory. The content of the CDN at https://cdn.proj.org +is https://github.com/OSGeo/PROJ-data , which has the same content +as https://github.com/OSGeo/proj-datumgrid converted in GeoTIFF files. In the +current state, we could have a somewhat inconsistency between users relying on +the proj-datumgrid, proj-datumgrid-[world,northamerica,oceania,europe] packages +of mostly NTv2 and GTX files, and what is shipped through the CDN. Maintaining +two repositories is also a maintenance burden in the long term.

+

It is thus desirable to have a single source of truth, and we propose it to be +based on the GeoTIFF grids.

+
+ +
+

Backward compatibility

+

This change is considered to be mostly backward compatible. There might be +impacts for software using proj_coordoperation_get_grid_used() and +assuming that the url returned is one of the proj-datumgrid-xxx files at +https://download.osgeo.org. As mentioned in +https://lists.osgeo.org/pipermail/proj/2020-January/009274.html , this +assumption was not completely bullet-proof either. +There will be impacts on software checking the value of PROJ pipeline strings +resulting proj_create_crs_to_crs(). The new grid names will now +be returned (the most impacted software will likely be PROJ’s own test suite)

+

Although discouraged, people not using the new proj-datumgrid-geotiff-XXX.zip +archives, should still be able to use the old archives made of NTv2/GTX files, +at least as long as the PROJ database does not only point to a GeoTIFF grid. +So this might be a short-term partly working solution, but at time goes, it +will become increasingly non-working. The nominal combination will be +PROJ 7.0 + proj-datumgrid-geotiff-1.0.zip

+
+
+

Testing

+

PROJ test suite will have to be adapted for the new TIFF based filenames.

+

Mechanism to auto-promote existing NTv2/GTX names to TIFF ones will be exercised.

+
+
+

Proposed implementation

+

https://github.com/OSGeo/PROJ/pull/1891 and https://github.com/OSGeo/PROJ-data/pull/5

+
+
+

Adoption status

+

The RFC was adopted on 2020-01-28 with +1’s from the following PSC members

+
    +

  • Kristian Evers

    • +

  • Even Rouault

    • +

  • Thomas Knudsen

    • +

  • Howard Butler

    • +

  • Kurt Schwehr

    • +
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/community/rfc/rfc-6.html b/community/rfc/rfc-6.html new file mode 100644 index 00000000..ed464a2e --- /dev/null +++ b/community/rfc/rfc-6.html @@ -0,0 +1,474 @@ + + + + + + + PROJ RFC 6: Triangulation-based transformations — PROJ 9.0.0 documentation + + + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

PROJ RFC 6: Triangulation-based transformations

+
+
Author
+

Even Rouault

+
+
Contact
+

even.rouault@spatialys.com

+
+
Status
+

Adopted

+
+
Implementation target
+

PROJ 7.2

+
+
Last Updated
+

2020-09-02

+
+
+
+

Summary

+

This RFC adds a new transformation method, tinshift (TIN stands for +Triangulated Irregular Network)

+

The motivation for this work is to be able to handle the official transformations +created by National Land Survey of Finland, for:

+
    +

  • horizontal transformation between the KKJ and ETRS89 horizontal datums

    • +

  • vertical transformations between N43 and N60 heights, and N60 and N2000 heights.

    • +
+

Such transformations are somehow related to traditional grid-based transformations, +except that the correction values are hold by the vertices of the triangulation, +instead of being at nodes of a grid.

+

Triangulation are in a number of cases the initial product of a geodesic adjustment, +with grids being a derived product. The Swiss grids have for example +derived products of an original triangulation.

+

Grid-based transformations remain very convenient to use because accessing +correction values is really easy and efficient, so triangulation-based transformations +are not meant as replacing them, but more about it being a complement, that is +sometimes necessary to be able to replicate the results of a officially vetted +transformation to a millimetric or better precision (speaking here about reproducibility +of numeric results, rather than the physical accuracy of the transformation that +might rather be centimetric). It is always possible to approach the result of +the triangulation with a grid, but that may require to adopt a small grid step, +and thus generate a grid that can be much larger than the original triangulation.

+
+
+

Details

+
+

Transformation

+

A new transformation method, tinshift, is added. It takes one mandatory +argument, file, that points to a JSON file, which contains the triangulation +and associated metadata. Input and output coordinates must be geographic or projected. +Depending on the content +of the JSON file, horizontal, vertical or both components of the coordinates may +be transformed.

+

The transformation is used like:

+
$ echo 3210000.0000 6700000.0000 0 2020 | cct +proj=tinshift +file=./triangulation_kkj.json
+
+209948.3217     6697187.0009    0.0000     2020
+
+
+

The transformation is invertible, with the same computational complexity than +the forward transformation.

+
+
+

Algorithm

+

Internally, tinshift ingest the whole file into memory. It is considered that +triangulation should be small enough for that. The above mentioned KKJ to ETRS89 +triangulation fits into 65 KB of JSON, for 1449 triangles and 767 vertices.

+

When a point is transformed, one must find the triangle into which it falls into. +Instead of iterating over all triangles, we build a in-memory quadtree to speed-up +the identification of candidates triangles. On the above mentioned KKJ -> ETRS89 +triangulation, this speeds up the whole transformation by a factor of 10. The +quadtree structure is a very good compromise between the performance gain it brings +and the simplicity of its implementation (we have ported the implementation coming +from GDAL, inherit from the one used for shapefile .spx spatial indices).

+

To determine if a point falls into a triangle, one computes its 3 +barycentric coordinates +from its projected coordinates, \(\lambda_i\) for \(i=1,2,3\). +They are real values (in the [0,1] range for a point inside the triangle), +giving the weight of each of the 3 vertices of the triangles.

+

Once those weights are known, interpolating the target horizontal +coordinate is a matter of doing the linear combination of those weights with +the target horizontal coordinates at the 3 vertices of the triangle (\(Xt_i\) and \(Yt_i\)):

+
+\[ \begin{align}\begin{aligned}X_{target} = Xt_1 * \lambda_1 + Xt_2 * \lambda_2 + Xt_3 * \lambda_3\\Y_{target} = Yt_1 * \lambda_1 + Yt_2 * \lambda_2 + Yt_3 * \lambda_3\end{aligned}\end{align} \]
+

This interpolation is exact at the vertices of the triangulation, and has linear properties +inside each triangle. It is completely equivalent to other formulations of +triangular interpolation, such as

+
+\[ \begin{align}\begin{aligned}X_{target} = A + X_{source} * B + Y_{source} * C\\Y_{target} = D + X_{source} * E + Y_{source} * F\end{aligned}\end{align} \]
+

where the A, B, C, D, E, F constants (for a given triangle) are found by solving +the 2 systems of 3 linear equations, constraint by the source and target coordinate pairs +of the 3 vertices of the triangle:

+
+\[ \begin{align}\begin{aligned}Xt_i = A + Xs_i * B + Ys_i * C\\Yt_i = D + Xs_i * E + Ys_i * F\end{aligned}\end{align} \]
+
+

Note

+

From experiments, the interpolation using barycentric coordinates is slightly +more numerically robust when interpolating projected coordinates of amplitude of the +order of 1e5 / 1e6, due to computations involving differences of coordinates. +Whereas the formulation with the A, B, C, D, E, F tends to have big values for +the A and D constants, and values clause to 0 for C and E, and close to 1 for +B and F. However, the difference between the two approaches is negligible for +practical purposes (below micrometre precision)

+
+

Similarly for a vertical coordinate transformation, where \(Zoff_i\) is the vertical +offset at each vertex of the triangle:

+
+\[Z_{target} = Z_{source} + Zoff_1 * \lambda_1 + Zoff_2 * \lambda_2 + Zoff_3 * \lambda_3\]
+
+
+

Constraints on the triangulation

+

No check is done on the consistence of the triangulation. It is highly +recommended that triangles do not overlap each other (when considering the +source coordinates or the forward transformation, or the target coordinates for +the inverse transformation), otherwise which triangle will be selected is +unspecified. Besides that, the triangulation does not need to have particular +properties (like being a Delaunay triangulation)

+
+
+

File format

+

To the best of our knowledge, there are no established file formats to convey +geodetic transformations as triangulations. Potential similar formats to store TINs +are ITF or +XMS. +Both of them would need to be extended in order to handle datum shift information, +since they are both intended for mostly DEM use.

+

We thus propose a text-based format, using JSON as a serialization. Using a text-based +format could potentially be thought as a limitation performance-wise compared to +binary formats, but for the size of triangulations considered (a few thousands triangles / vertices), +there is no issue. Loading such file is a matter of 20 milliseconds or so. For reference, +loading a triangulation of about 115 000 triangles and 71 000 vertices takes 450 ms.

+

Using JSON provides generic formatting and parsing rules, and convenience to +create it from Python script for examples. This could also be easily generated “at hand” +by non-JSON aware writers.

+

For generic metadata, we reuse closely what has been used for the +Deformation model master file

+

Below a minimal example, from the KKJ to ETRS89 transformation, with just a +single triangle:

+
{
+    "file_type": "triangulation_file",
+    "format_version": "1.0",
+    "name": "Name",
+    "version": "Version",
+    "publication_date": "2018-07-01T00:00:00Z",
+    "license": "Creative Commons Attribution 4.0 International",
+    "description": "Test triangulation",
+    "authority": {
+        "name": "Authority name",
+        "url": "http://example.com",
+        "address": "Address",
+        "email": "test@example.com"
+    },
+    "links": [
+    {
+        "href": "https://example.com/about.html",
+        "rel": "about",
+        "type": "text/html",
+        "title": "About"
+    },
+    {
+        "href": "https://example.com/download",
+        "rel": "source",
+        "type": "application/zip",
+        "title": "Authoritative source"
+    },
+    {
+        "href": "https://creativecommons.org/licenses/by/4.0/",
+        "rel": "license",
+        "type": "text/html",
+        "title": "Creative Commons Attribution 4.0 International license"
+    },
+    {
+        "href": "https://example.com/metadata.xml",
+        "rel": "metadata",
+        "type": "application/xml",
+    "title": " ISO 19115 XML encoded metadata regarding the triangulation"
+        }
+    ],
+    "input_crs": "EPSG:2393",
+    "target_crs": "EPSG:3067",
+    "transformed_components": [ "horizontal" ],
+    "vertices_columns": [ "source_x", "source_y", "target_x", "target_y" ],
+    "triangles_columns": [ "idx_vertex1", "idx_vertex2", "idx_vertex3" ],
+    "vertices": [ [3244102.707, 6693710.937, 244037.137, 6690900.686],
+                  [3205290.722, 6715311.822, 205240.895, 6712492.577],
+                  [3218328.492, 6649538.429, 218273.648, 6646745.973] ],
+    "triangles": [ [0, 1, 2] ]
+}
+
+
+

So after the generic metadata, we define the input and output CRS (informative +only), and that the transformation affects horizontal components of +coordinates. We name the columns of the vertices and triangles arrays. +We defined the source and target coordinates of each vertex, and define a +triangle by referring to the index of its vertices in the vertices array.

+

More formally, the specific items for the triangulation file are:

+
+
input_crs

String identifying the CRS of source coordinates +in the vertices. Typically EPSG:XXXX. If the transformation is for vertical +component, this should be the code for a compound CRS (can be EPSG:XXXX+YYYY +where XXXX is the code of the horizontal CRS and YYYY the code of the vertical CRS). +For example, for the KKJ->ETRS89 transformation, this is EPSG:2393 +(KKJ / Finland Uniform Coordinate System). The input coordinates are assumed +to be passed in the “normalized for visualisation” / “GIS friendly” order, +that is longitude, latitude for geographic coordinates and +easting, northing for projected coordinates.

+
+
output_crs

String identifying the CRS of target coordinates in the vertices. +Typically EPSG:XXXX. If the transformation is for vertical component, +this should be the code for a compound CRS (can be EPSG:XXXX+YYYY where +XXXX is the code of the horizontal CRS and YYYY the code of the vertical CRS). +For example, for the KKJ->ETRS89 transformation, this is EPSG:3067 +("ETRS89 / TM35FIN(E,N)"). The output coordinates will be returned in +the “normalized for visualisation” / “GIS friendly” order, +that is longitude, latitude for geographic coordinates and +easting, northing for projected coordinates.

+
+
transformed_components

Array which may contain one or two strings: “horizontal” when horizontal +components of the coordinates are transformed and/or “vertical” when the +vertical component is transformed.

+
+
vertices_columns

Specify the name of the columns of the rows in the vertices +array. There must be exactly as many elements in vertices_columns as in a +row of vertices. The following names have a special meaning: source_x, +source_y, target_x, target_y, source_z, target_z and +offset_z. source_x and source_y are compulsory. +source_x is for the source longitude (in degree) or easting. +source_y is for the source latitude (in degree) or northing. +target_x and target_y are compulsory when horizontal is specified +in transformed_components. (source_z and target_z) or +offset_z are compulsory when vertical is specified in transformed_components

+
+
triangles_columns

Specify the name of the columns of the rows in the +triangles array. There must be exactly as many elements in triangles_columns +as in a row of triangles. The following names have a special meaning: +idx_vertex1, idx_vertex2, idx_vertex3. They are compulsory.

+
+
vertices

An array whose items are themselves arrays with as many columns as +described in vertices_columns.

+
+
triangles

An array whose items are themselves arrays with as many columns as +described in triangles_columns. +The value of the idx_vertexN columns must be indices +(between 0 and len(vertices-1) of items of the vertices array.

+
+
+
+
+

Code impacts

+

The following new files are added in src/transformations:

+
    +

  • tinshift.cpp: PROJ specific code for defining the new operation. Takes care +of the input and output coordinate conversions (between input_crs and triangulation_source_crs, +and triangulation_target_crs and output_crs), when needed.

    • +

  • tinshift.hpp: Header-based implementation. This file contains the API.

    • +

  • tinshift_exceptions.hpp: Exceptions that can be raised during file parsing

    • +

  • tinshift_impl.hpp: Implementation of file loading, triangle search and interpolation.

    • +
+

This is the approach that has been followed for the deformation model implementation, +and which makes it easier to do unit test.

+

src/quadtree.hpp contains a quadtree implementation.

+
+
+

Performance indications

+

Tested on Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz, transforming 4 million points

+

For the KKJ to ETRS89 transformation (1449 triangles and 767 vertices), +4.4 million points / sec can be transformed.

+

For comparison, the Helmert-based KKJ to ETRS89 transformation operates at +1.6 million points / sec.

+

A triangulation with about 115 000 triangles and 71 000 vertices +operates at 2.2 million points / sec +(throughput on more points would be better since the initial loading of the +triangulation is non-negligible here)

+
+
+
+

Backward compatibility

+

New functionality fully backward compatible.

+
+
+

Testing

+

The PROJ test suite will be enhanced to test the new transformation, with a +new .gie file, and a C++ unit test to test at a lower level.

+
+
+

Documentation

+ +
+
+

Proposed implementation

+

An initial implementation is available at https://github.com/rouault/PROJ/tree/tinshift

+
+
+

References

+

Finnish coordinate transformation (automated translation to English)

+
+
+

Adoption status

+

The RFC was adopted on 2020-09-02 with +1’s from the following PSC members

+
    +

  • Kristian Evers

    • +

  • Charles Karney

    • +

  • Thomas Knudsen

    • +

  • Even Rouault

    • +
+
+
+

Funding

+

This work is funded by National Land Survey of Finland

+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file diff --git a/community/rfc/rfc-7.html b/community/rfc/rfc-7.html new file mode 100644 index 00000000..b3dab5f5 --- /dev/null +++ b/community/rfc/rfc-7.html @@ -0,0 +1,294 @@ + + + + + + + PROJ RFC 7: Drop Autotools, maintain CMake — PROJ 9.0.0 documentation + + + + + + + + + + + + + + + + + + + +
+ + +
+ +
+
+
+ +
+
+
+
+ +
+

PROJ RFC 7: Drop Autotools, maintain CMake

+
+
Author
+

Mike Taves

+
+
Contact
+

mwtoews@gmail.com

+
+
Status
+

Adopted

+
+
Implementation target
+

PROJ 9.0

+
+
Last Updated
+

2021-10-27

+
+
+
+

Summary

+

This RFC proposes to drop Autotools for PROJ 9.0, and to maintain CMake +for build automation, testing and packaging. This will reduce the overall +maintenance for PROJ and will enable the library to be integrated into other +projects that use CMake.

+
+
+

Background

+

Here is a short timeline of the build tools used for PROJ:

+
    +

  • Throughout the mid-1990s, Gerald Evenden maintained a Unix build system with +a few scripts (some derived from Autoconf), and Makefile templates.

    • +

  • In 2000, Frank Warmerdam wrote Autoconf and Automake configurations for +PROJ 4.4.0.

    • +

  • This was followed by a NMake configuration to build PROJ 4.4.2 for Windows.

    • +

  • In 2014, a CMake build setup was introduced by Howard Butler for +PROJ 4.9.0RC1. The CMake configuration was improved for the 4.9.1 release, +but not considered at feature parity with the Autotools builds at the time.

    • +

  • The NMake build setup was removed for PROJ 6.0.0, as its functionality had +been replaced by CMake.

    • +
+
+
+

Motivation

+

The primary motivation in removing Autotools is to reduce the burden of +maintaining multiple build configurations, which requires developers to be +familiar with different tools and configuration files. There are several other +benefits in maintaining a single build system:

+
    +

  • Remove extra configuration and m4 macro files from source repository,

    • +

  • Simplify scripts used for running tests for CI services (GitHub Actions, +TravisCI),

    • +

  • Reduce compilation time (and carbon footprint) used for testing on CI +services,

    • +

  • Ease development effort, particularly with new contributors.

    • +
+
+
+

Why drop Autotools?

+

The GNU Build System or Autotools consist of a suite of tools including +Autoconf and Automake, which can be used to build software on Unix-like +systems. These tools are not cross-platform, and do not naively integrate +with development environments on Microsoft Windows. Furthermore, the existing +PROJ Autotools builds do not install the CMake configuration files required to +find PROJ from other projects that use CMake +(#2546).

+
+
+

Why use CMake?

+

CMake is an open source cross-platform tool for build automation, testing and +packaging of software. It does not directly compile the software, but manages +the build process using generators, including Unix Makefiles and Ninja among +other command-based and IDE tools. The CMake software has been under active +development since its origins in 2000. The CMake language is carefully +developed with backwards-compatible policies that aim to provide consistent +behaviour across different versions. CMake is currently the preferred build +tool for PROJ for the following reasons:

+
    +

  • It has existed in the PROJ code base since 2014, and is familiar to active +PROJ contributors,

    • +

  • It can install configuration files that can be used by other software that +use CMake to find PROJ for linking via find_package(),

    • +

  • CMake configurations are used in 3rd-party binary packages of PROJ, +including conda-forge and vcpkg,

    • +

  • It can be used to build PROJ on all major operating systems and compiler +combinations (where compatible),

    • +

  • It has integration with modern IDEs and tools, including +Microsoft Visual Studio and Qt Creator.

    • +
+
+
+

Why not CMake?

+

Other modern cross-platform build systems exist, including Meson and Bazel, +which have many advantages over CMake. However, they are currently not widely +used by active PROJ contributors. This RFC should not restrict future build +system configurations being introduced to PROJ, if they are proven to have +advantages to CMake over time.

+
+
+

Potential impacts

+

Binary packagers that currently rely on Autotools would obviously need to +transition building and testing PROJ with CMake. Issues related to +multiarch builds of PROJ may become apparent, which can be patched and/or +reported to PROJ developers. One feature of Autotools is that both static and +dynamic (shared) libraries are built, which packagers may distribute. This +feature is currently not set-up for PROJ, as it would need to be configured +and built twice.

+

End-users that use binary packages of PROJ should not be impacted. PROJ should +be discoverable via both pkg-config and CMake’s find_package(). +Other projects that use Autotools will continue to work as expected, +linking statically or dynamically to PROJ built by CMake.

+
+
+

Transition plan

+

If this proposal is approved, the following tasks should be completed:

+
    +

  • Rewrite CI tests to only use CMake for packaging, building, testing, +installation and post-install tests,

    • +

  • Remove files only used by Autotools, also update .gitignore,

    • +

  • Update documentation and HOWTORELEASE notes.

    • +
+

Related issues will be tracked on GitHub with a tag +RFC7: Autotools→CMake.

+
+
+

Adoption status

+

The RFC was adopted on 2021-10-26 with +1’s from the following PSC members

+
    +

  • Kristian Evers

    • +

  • Even Rouault

    • +

  • Howard Butler

    • +

  • Thomas Knudsen

    • +

  • Kurt Schwehr

    • +

  • Charles Karney

    • +

  • Thomas Knudsen

    • +
+
+
+ + +
+
+ +
+
+
+
+ + + + \ No newline at end of file -- cgit v1.2.3