diff options
| author | Billy O'Neal <bion@microsoft.com> | 2021-04-26 10:27:45 -0700 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2021-04-26 10:27:45 -0700 |
| commit | a9b27ed5dffbf70b135eddb0c4729f3ca87f106c (patch) | |
| tree | cc562b219c82e40567925a48248cbc17b2cc2b59 /scripts | |
| parent | 84e5852cfaa8e7875027e79a29a5534bb5d4a475 (diff) | |
| download | vcpkg-a9b27ed5dffbf70b135eddb0c4729f3ca87f106c.tar.gz vcpkg-a9b27ed5dffbf70b135eddb0c4729f3ca87f106c.zip | |
Update VMs, CMake to 3.20.1, CUDA to 11.3, and pwsh to 7.1.3 (#17331)
* Update to CUDA 11.3 on Windows.
* Update PowerShell to 7.1.3.
* Update CUDA to 11.3 on Linux.
* "Explode" VM provisioning scripts for more consistent feedback during deploy. This resolves the deployment script often hanging with no feedback as to why.
* [libdatachannel] Fix use of deprecated result_type typedef.
Submitted upstream as https://github.com/paullouisageneau/libdatachannel/pull/413
* [libvpx] Get the libvpx outputs from the correct place.
(Perhaps VS2019 version 16.10 moved where these are placed? I've been defensive and left an attempt to find from the old location in place.)
* [chromaprint] Support implementations where lrintf is an intrinsic.
* Add provision-entire-image script.
* [cudnn] Disable download-on-the-fly due to licensing concerns.
* Add libnccl to Linux VMs.
* [wangle] Disable x64-windows due to an ICE.
* [cmake] Update cmake to 3.20.1 to avoid https://gitlab.kitware.com/cmake/cmake/-/issues/21571 race
* [libudis86] Fix passing a bogus working directory which fails on CMake 3.20.x
* [dartsim] Disable unit tests, examples, and tutorials, some of which have CMake authoring errors rejected by 3.20.1.
* Add thrust to the cuda installees.
* [tensorflow] Put .bzl in CURRENT_BUILDTREES_DIR to avoid running out of disk space in CI and to respect --clean-after-build.
* [dimcli] Skip port broken by changes in VS2019 project system.
* [upb] Disable an additional warning.
* [libhv] Fix typo DISABLE_PARALLEL => DISABLE_PARALLEL_CONFIGURE
* Update pools
Diffstat (limited to 'scripts')
| -rw-r--r-- | scripts/azure-pipelines/azure-pipelines.yml | 4 | ||||
| -rwxr-xr-x | scripts/azure-pipelines/linux/provision-image.sh | 13 | ||||
| -rw-r--r-- | scripts/azure-pipelines/windows/create-vmss.ps1 | 247 | ||||
| -rw-r--r-- | scripts/azure-pipelines/windows/deploy-cuda.ps1 | 62 | ||||
| -rw-r--r-- | scripts/azure-pipelines/windows/deploy-mpi.ps1 | 47 | ||||
| -rw-r--r-- | scripts/azure-pipelines/windows/deploy-psexec.ps1 | 8 | ||||
| -rw-r--r-- | scripts/azure-pipelines/windows/deploy-pwsh.ps1 | 9 | ||||
| -rw-r--r-- | scripts/azure-pipelines/windows/deploy-settings.txt | 20 | ||||
| -rw-r--r-- | scripts/azure-pipelines/windows/deploy-visual-studio.ps1 | 86 | ||||
| -rw-r--r-- | scripts/azure-pipelines/windows/deploy-windows-wdk.ps1 | 47 | ||||
| -rw-r--r-- | scripts/azure-pipelines/windows/drop-to-admin-user-prefix.ps1 | 27 | ||||
| -rw-r--r-- | scripts/azure-pipelines/windows/provision-entire-image.ps1 | 16 | ||||
| -rw-r--r-- | scripts/azure-pipelines/windows/provision-image.txt | 454 | ||||
| -rw-r--r-- | scripts/azure-pipelines/windows/utility-prefix.ps1 | 125 | ||||
| -rw-r--r-- | scripts/ci.baseline.txt | 13 | ||||
| -rw-r--r-- | scripts/vcpkgTools.xml | 38 |
16 files changed, 676 insertions, 540 deletions
diff --git a/scripts/azure-pipelines/azure-pipelines.yml b/scripts/azure-pipelines/azure-pipelines.yml index 27db6a93d..9f6526cb7 100644 --- a/scripts/azure-pipelines/azure-pipelines.yml +++ b/scripts/azure-pipelines/azure-pipelines.yml @@ -2,8 +2,8 @@ # SPDX-License-Identifier: MIT
#
variables:
- windows-pool: 'PrWin-2020-12-17'
- linux-pool: 'PrLin-2020-12-16'
+ windows-pool: 'PrWin-2021-04-23'
+ linux-pool: 'PrLin-2021-04-25'
osx-pool: 'PrOsx-2020-09-28'
stages:
diff --git a/scripts/azure-pipelines/linux/provision-image.sh b/scripts/azure-pipelines/linux/provision-image.sh index c04f4b3ca..6663baed8 100755 --- a/scripts/azure-pipelines/linux/provision-image.sh +++ b/scripts/azure-pipelines/linux/provision-image.sh @@ -58,14 +58,13 @@ cd ~ curl -sSL https://get.haskellstack.org/ | sudo sh # Install CUDA -wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.2.89-1_amd64.deb -sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub -sudo dpkg -i cuda-repo-ubuntu1804_10.2.89-1_amd64.deb -wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb -sudo dpkg -i nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb +wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin +sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600 +sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub +sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /" sudo apt -y update -sudo apt install -y --no-install-recommends cuda-compiler-10-2 cuda-libraries-dev-10-2 cuda-driver-dev-10-2 cuda-cudart-dev-10-2 libcublas10 cuda-curand-dev-10-2 -sudo apt install -y --no-install-recommends libcudnn7-dev +sudo apt install -y --no-install-recommends cuda-compiler-11-3 cuda-libraries-dev-11-3 cuda-driver-dev-11-3 \ + cuda-cudart-dev-11-3 libcublas-11-3 libcurand-dev-11-3 libcudnn8-dev libnccl2 libnccl-dev # Install PowerShell wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb diff --git a/scripts/azure-pipelines/windows/create-vmss.ps1 b/scripts/azure-pipelines/windows/create-vmss.ps1 index 608465100..fddba0eec 100644 --- a/scripts/azure-pipelines/windows/create-vmss.ps1 +++ b/scripts/azure-pipelines/windows/create-vmss.ps1 @@ -21,11 +21,18 @@ the compiler rather than for testing vcpkg. Differences: * The machine prefix is changed to VcpkgUnstable instead of PrWin.
* No storage account or "archives" share is provisioned.
* The firewall is not opened to allow communication with Azure Storage.
+
+.PARAMETER CudnnPath
+The path to a CUDNN zip file downloaded from NVidia official sources
+(e.g. https://developer.nvidia.com/compute/machine-learning/cudnn/secure/8.1.1.33/11.2_20210301/cudnn-11.2-windows-x64-v8.1.1.33.zip
+downloaded in a browser with an NVidia account logged in.)
#>
[CmdLetBinding()]
Param(
- [switch]$Unstable = $false
+ [switch]$Unstable = $false,
+ [parameter(Mandatory=$true)]
+ [string]$CudnnPath
)
$Location = 'westus2'
@@ -43,15 +50,16 @@ $WindowsServerSku = '2019-Datacenter' $ErrorActionPreference = 'Stop'
$ProgressActivity = 'Creating Scale Set'
-$TotalProgress = 12
-if ($Unstable) {
- $TotalProgress -= 1 # skipping the archives share part
-}
-
+$TotalProgress = 18
$CurrentProgress = 1
Import-Module "$PSScriptRoot/../create-vmss-helpers.psm1" -DisableNameChecking
+if (-Not $CudnnPath.EndsWith('.zip')) {
+ Write-Error 'Expected CudnnPath to be a zip file.'
+ return
+}
+
####################################################################################################
Write-Progress `
-Activity $ProgressActivity `
@@ -155,57 +163,84 @@ $VirtualNetwork = New-AzVirtualNetwork ` -Subnet $Subnet
####################################################################################################
-if (-Not $Unstable) {
- Write-Progress `
- -Activity $ProgressActivity `
- -Status 'Creating archives storage account' `
- -PercentComplete (100 / $TotalProgress * $CurrentProgress++)
+Write-Progress `
+ -Activity $ProgressActivity `
+ -Status 'Creating storage account' `
+ -CurrentOperation 'Initial setup' `
+ -PercentComplete (100 / $TotalProgress * $CurrentProgress++)
- $StorageAccountName = Sanitize-Name $ResourceGroupName
+$StorageAccountName = Sanitize-Name $ResourceGroupName
- New-AzStorageAccount `
- -ResourceGroupName $ResourceGroupName `
- -Location $Location `
- -Name $StorageAccountName `
- -SkuName 'Standard_LRS' `
- -Kind StorageV2
+New-AzStorageAccount `
+ -ResourceGroupName $ResourceGroupName `
+ -Location $Location `
+ -Name $StorageAccountName `
+ -SkuName 'Standard_LRS' `
+ -Kind StorageV2
- $StorageAccountKeys = Get-AzStorageAccountKey `
- -ResourceGroupName $ResourceGroupName `
- -Name $StorageAccountName
+$StorageAccountKeys = Get-AzStorageAccountKey `
+ -ResourceGroupName $ResourceGroupName `
+ -Name $StorageAccountName
- $StorageAccountKey = $StorageAccountKeys[0].Value
+$StorageAccountKey = $StorageAccountKeys[0].Value
- $StorageContext = New-AzStorageContext `
- -StorageAccountName $StorageAccountName `
- -StorageAccountKey $StorageAccountKey
+$StorageContext = New-AzStorageContext `
+ -StorageAccountName $StorageAccountName `
+ -StorageAccountKey $StorageAccountKey
- New-AzStorageContainer -Name archives -Context $StorageContext -Permission Off
- $StartTime = [DateTime]::Now
- $ExpiryTime = $StartTime.AddMonths(6)
+Write-Progress `
+ -Activity $ProgressActivity `
+ -Status 'Creating storage account' `
+ -CurrentOperation 'Uploading cudnn.zip' `
+ -PercentComplete (100 / $TotalProgress * $CurrentProgress) # note no ++
- $SasToken = New-AzStorageAccountSASToken `
- -Service Blob `
- -Permission "racwdlup" `
- -Context $StorageContext `
- -StartTime $StartTime `
- -ExpiryTime $ExpiryTime `
- -ResourceType Service,Container,Object `
- -Protocol HttpsOnly
+New-AzStorageContainer -Name setup -Context $storageContext -Permission blob
- $SasToken = $SasToken.Substring(1) # strip leading ?
+Set-AzStorageBlobContent -File $CudnnPath `
+ -Container 'setup' `
+ -Blob 'cudnn.zip' `
+ -Context $StorageContext
- # Note that we put the storage account into the firewall after creating the above SAS token or we
- # would be denied since the person running this script isn't one of the VMs we're creating here.
- Set-AzStorageAccount `
- -ResourceGroupName $ResourceGroupName `
- -AccountName $StorageAccountName `
- -NetworkRuleSet ( `
- @{bypass="AzureServices"; `
- virtualNetworkRules=( `
- @{VirtualNetworkResourceId=$VirtualNetwork.Subnets[0].Id;Action="allow"}); `
- defaultAction="Deny"})
-}
+$CudnnBlobUrl = "https://$StorageAccountName.blob.core.windows.net/setup/cudnn.zip"
+
+Write-Progress `
+ -Activity $ProgressActivity `
+ -Status 'Creating storage account' `
+ -CurrentOperation 'Creating archives container' `
+ -PercentComplete (100 / $TotalProgress * $CurrentProgress) # note no ++
+
+New-AzStorageContainer -Name archives -Context $StorageContext -Permission Off
+
+$StartTime = [DateTime]::Now
+$ExpiryTime = $StartTime.AddMonths(6)
+
+$SasToken = New-AzStorageAccountSASToken `
+ -Service Blob `
+ -Permission "racwdlup" `
+ -Context $StorageContext `
+ -StartTime $StartTime `
+ -ExpiryTime $ExpiryTime `
+ -ResourceType Service,Container,Object `
+ -Protocol HttpsOnly
+
+$SasToken = $SasToken.Substring(1) # strip leading ?
+
+Write-Progress `
+ -Activity $ProgressActivity `
+ -Status 'Creating storage account' `
+ -CurrentOperation 'Locking down network' `
+ -PercentComplete (100 / $TotalProgress * $CurrentProgress) # note no ++
+
+# Note that we put the storage account into the firewall after creating the above SAS token or we
+# would be denied since the person running this script isn't one of the VMs we're creating here.
+Set-AzStorageAccount `
+ -ResourceGroupName $ResourceGroupName `
+ -AccountName $StorageAccountName `
+ -NetworkRuleSet ( `
+ @{bypass="AzureServices"; `
+ virtualNetworkRules=( `
+ @{VirtualNetworkResourceId=$VirtualNetwork.Subnets[0].Id;Action="allow"}); `
+ defaultAction="Deny"})
####################################################################################################
Write-Progress `
@@ -245,30 +280,128 @@ New-AzVm ` ####################################################################################################
Write-Progress `
-Activity $ProgressActivity `
- -Status 'Running provisioning script provision-image.txt (as a .ps1) in VM' `
+ -Status 'Running provisioning script deploy-psexec.ps1 in VM' `
-PercentComplete (100 / $TotalProgress * $CurrentProgress++)
-$provisionParameters = @{AdminUserPassword = $AdminPW;}
-if (-Not $Unstable) {
- $provisionParameters['StorageAccountName'] = $StorageAccountName
- $provisionParameters['StorageAccountSasToken'] = $SasToken
+$DeployPsExecResult = Invoke-AzVMRunCommand `
+ -ResourceGroupName $ResourceGroupName `
+ -VMName $ProtoVMName `
+ -CommandId 'RunPowerShellScript' `
+ -ScriptPath "$PSScriptRoot\deploy-psexec.ps1"
+
+Write-Host "deploy-psexec.ps1 output: $($DeployPsExecResult.value.Message)"
+
+####################################################################################################
+function Invoke-ScriptWithPrefix {
+ param(
+ [string]$ScriptName,
+ [switch]$AddAdminPw,
+ [switch]$AddCudnnUrl
+ )
+
+ Write-Progress `
+ -Activity $ProgressActivity `
+ -Status "Running provisioning script $ScriptName in VM" `
+ -PercentComplete (100 / $TotalProgress * $CurrentProgress++)
+
+ $DropToAdminUserPrefix = Get-Content "$PSScriptRoot\drop-to-admin-user-prefix.ps1" -Encoding utf8NoBOM -Raw
+ $UtilityPrefixContent = Get-Content "$PSScriptRoot\utility-prefix.ps1" -Encoding utf8NoBOM -Raw
+
+ $tempScriptFilename = [System.IO.Path]::GetTempPath() + [System.IO.Path]::GetRandomFileName() + ".txt"
+ try {
+ $script = Get-Content "$PSScriptRoot\$ScriptName" -Encoding utf8NoBOM -Raw
+ if ($AddAdminPw) {
+ $script = $script.Replace('# REPLACE WITH DROP-TO-ADMIN-USER-PREFIX.ps1', $DropToAdminUserPrefix)
+ }
+
+ if ($AddCudnnUrl) {
+ $script = $script.Replace('# REPLACE WITH $CudnnUrl', "`$CudnnUrl = '$CudnnBlobUrl'")
+ }
+
+ $script = $script.Replace('# REPLACE WITH UTILITY-PREFIX.ps1', $UtilityPrefixContent);
+ Set-Content -Path $tempScriptFilename -Value $script -Encoding utf8NoBOM
+
+ $parameter = $null
+ if ($AddAdminPw) {
+ $parameter = @{AdminUserPassword = $AdminPW;}
+ }
+
+ $InvokeResult = Invoke-AzVMRunCommand `
+ -ResourceGroupName $ResourceGroupName `
+ -VMName $ProtoVMName `
+ -CommandId 'RunPowerShellScript' `
+ -ScriptPath $tempScriptFilename `
+ -Parameter $parameter
+
+ Write-Host "$ScriptName output: $($InvokeResult.value.Message)"
+ } finally {
+ Remove-Item $tempScriptFilename -Force
+ }
}
+Invoke-ScriptWithPrefix -ScriptName 'deploy-visual-studio.ps1' -AddAdminPw
+Restart-AzVM -ResourceGroupName $ResourceGroupName -Name $ProtoVMName
+
+####################################################################################################
+Invoke-ScriptWithPrefix -ScriptName 'deploy-windows-wdk.ps1' -AddAdminPw
+Restart-AzVM -ResourceGroupName $ResourceGroupName -Name $ProtoVMName
+
+####################################################################################################
+Invoke-ScriptWithPrefix -ScriptName 'deploy-mpi.ps1' -AddAdminPw
+Restart-AzVM -ResourceGroupName $ResourceGroupName -Name $ProtoVMName
+
+####################################################################################################
+Invoke-ScriptWithPrefix -ScriptName 'deploy-cuda.ps1' -AddAdminPw -AddCudnnUrl
+Restart-AzVM -ResourceGroupName $ResourceGroupName -Name $ProtoVMName
+
+####################################################################################################
+Invoke-ScriptWithPrefix -ScriptName 'deploy-pwsh.ps1' -AddAdminPw
+Restart-AzVM -ResourceGroupName $ResourceGroupName -Name $ProtoVMName
+
+####################################################################################################
+Write-Progress `
+ -Activity $ProgressActivity `
+ -Status 'Running provisioning script deploy-settings.txt (as a .ps1) in VM' `
+ -PercentComplete (100 / $TotalProgress * $CurrentProgress++)
+
$ProvisionImageResult = Invoke-AzVMRunCommand `
-ResourceGroupName $ResourceGroupName `
-VMName $ProtoVMName `
-CommandId 'RunPowerShellScript' `
- -ScriptPath "$PSScriptRoot\provision-image.txt" `
- -Parameter $provisionParameters
+ -ScriptPath "$PSScriptRoot\deploy-settings.txt"
-Write-Host "provision-image.ps1 output: $($ProvisionImageResult.value.Message)"
+Write-Host "deploy-settings.txt output: $($ProvisionImageResult.value.Message)"
####################################################################################################
Write-Progress `
-Activity $ProgressActivity `
- -Status 'Restarting VM' `
+ -Status 'Deploying SAS token into VM' `
-PercentComplete (100 / $TotalProgress * $CurrentProgress++)
+$tempScriptFilename = [System.IO.Path]::GetTempPath() + [System.IO.Path]::GetRandomFileName() + ".txt"
+try {
+ $script = "Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\Environment' " `
+ + "-Name PROVISIONED_AZURE_STORAGE_NAME " `
+ + "-Value '$StorageAccountName'`r`n" `
+ + "Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\Environment' " `
+ + "-Name PROVISIONED_AZURE_STORAGE_SAS_TOKEN " `
+ + "-Value '$SasToken'`r`n"
+
+ Write-Host "Script content is:"
+ Write-Host $script
+
+ Set-Content -Path $tempScriptFilename -Value $script -Encoding utf8NoBOM
+ $InvokeResult = Invoke-AzVMRunCommand `
+ -ResourceGroupName $ResourceGroupName `
+ -VMName $ProtoVMName `
+ -CommandId 'RunPowerShellScript' `
+ -ScriptPath $tempScriptFilename
+
+ Write-Host "Deploy SAS token output: $($InvokeResult.value.Message)"
+} finally {
+ Remove-Item $tempScriptFilename -Force
+}
+
Restart-AzVM -ResourceGroupName $ResourceGroupName -Name $ProtoVMName
####################################################################################################
diff --git a/scripts/azure-pipelines/windows/deploy-cuda.ps1 b/scripts/azure-pipelines/windows/deploy-cuda.ps1 new file mode 100644 index 000000000..ab1cc1c54 --- /dev/null +++ b/scripts/azure-pipelines/windows/deploy-cuda.ps1 @@ -0,0 +1,62 @@ +# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: MIT
+
+# REPLACE WITH DROP-TO-ADMIN-USER-PREFIX.ps1
+
+# REPLACE WITH UTILITY-PREFIX.ps1
+
+# REPLACE WITH $CudnnUrl
+
+$CudnnLocalZipPath = "$PSScriptRoot\cudnn-11.2-windows-x64-v8.1.1.33.zip"
+
+$CudaUrl = 'https://developer.download.nvidia.com/compute/cuda/11.3.0/network_installers/cuda_11.3.0_win10_network.exe'
+
+$CudaFeatures = 'nvcc_11.3 cuobjdump_11.3 nvprune_11.3 cupti_11.3 memcheck_11.3 nvdisasm_11.3 nvprof_11.3 ' + `
+ 'visual_studio_integration_11.3 visual_profiler_11.3 visual_profiler_11.3 cublas_11.3 cublas_dev_11.3 ' + `
+ 'cudart_11.3 cufft_11.3 cufft_dev_11.3 curand_11.3 curand_dev_11.3 cusolver_11.3 cusolver_dev_11.3 ' + `
+ 'cusparse_11.3 cusparse_dev_11.3 npp_11.3 npp_dev_11.3 nvrtc_11.3 nvrtc_dev_11.3 nvml_dev_11.3 ' + `
+ 'occupancy_calculator_11.3 thrust_11.3 '
+
+$destination = "$env:ProgramFiles\NVIDIA GPU Computing Toolkit\CUDA\v11.3"
+
+try {
+ Write-Host 'Downloading CUDA...'
+ [string]$installerPath = Get-TempFilePath -Extension 'exe'
+ curl.exe -L -o $installerPath -s -S $CudaUrl
+ Write-Host 'Installing CUDA...'
+ $proc = Start-Process -FilePath $installerPath -ArgumentList @('-s ' + $CudaFeatures) -Wait -PassThru
+ $exitCode = $proc.ExitCode
+ if ($exitCode -eq 0) {
+ Write-Host 'Installation successful!'
+ }
+ else {
+ Write-Error "Installation failed! Exited with $exitCode."
+ throw
+ }
+}
+catch {
+ Write-Error "Failed to install CUDA! $($_.Exception.Message)"
+ throw
+}
+
+try {
+ if ([string]::IsNullOrWhiteSpace($CudnnUrl)) {
+ if (-Not (Test-Path $CudnnLocalZipPath)) {
+ throw "CUDNN zip ($CudnnLocalZipPath) was missing, please download from NVidia and place next to this script."
+ }
+
+ $cudnnZipPath = $CudnnLocalZipPath
+ } else {
+ Write-Host 'Downloading CUDNN...'
+ $cudnnZipPath = Get-TempFilePath -Extension 'zip'
+ curl.exe -L -o $cudnnZipPath -s -S $CudnnUrl
+ }
+
+ Write-Host "Installing CUDNN to $destination..."
+ tar.exe -xvf "$cudnnZipPath" --strip 1 --directory "$destination"
+ Write-Host 'Installation successful!'
+}
+catch {
+ Write-Error "Failed to install CUDNN! $($_.Exception.Message)"
+ throw
+}
diff --git a/scripts/azure-pipelines/windows/deploy-mpi.ps1 b/scripts/azure-pipelines/windows/deploy-mpi.ps1 new file mode 100644 index 000000000..ba369f797 --- /dev/null +++ b/scripts/azure-pipelines/windows/deploy-mpi.ps1 @@ -0,0 +1,47 @@ +# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: MIT
+
+# REPLACE WITH DROP-TO-ADMIN-USER-PREFIX.ps1
+
+# REPLACE WITH UTILITY-PREFIX.ps1
+
+$MpiUrl = 'https://download.microsoft.com/download/a/5/2/a5207ca5-1203-491a-8fb8-906fd68ae623/msmpisetup.exe'
+
+<#
+.SYNOPSIS
+Installs MPI
+
+.DESCRIPTION
+Downloads the MPI installer located at $Url, and installs it with the
+correct flags.
+
+.PARAMETER Url
+The URL of the installer.
+#>
+Function InstallMpi {
+ Param(
+ [String]$Url
+ )
+
+ try {
+ Write-Host 'Downloading MPI...'
+ [string]$installerPath = Get-TempFilePath -Extension 'exe'
+ curl.exe -L -o $installerPath -s -S $Url
+ Write-Host 'Installing MPI...'
+ $proc = Start-Process -FilePath $installerPath -ArgumentList @('-force', '-unattend') -Wait -PassThru
+ $exitCode = $proc.ExitCode
+ if ($exitCode -eq 0) {
+ Write-Host 'Installation successful!'
+ }
+ else {
+ Write-Error "Installation failed! Exited with $exitCode."
+ throw
+ }
+ }
+ catch {
+ Write-Error "Failed to install MPI! $($_.Exception.Message)"
+ throw
+ }
+}
+
+InstallMpi -Url $MpiUrl
diff --git a/scripts/azure-pipelines/windows/deploy-psexec.ps1 b/scripts/azure-pipelines/windows/deploy-psexec.ps1 new file mode 100644 index 000000000..2a1f6fb14 --- /dev/null +++ b/scripts/azure-pipelines/windows/deploy-psexec.ps1 @@ -0,0 +1,8 @@ +# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: MIT
+
+$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue'
+$PsExecPath = 'C:\PsExec64.exe'
+Write-Host "Downloading psexec to: $PsExecPath"
+& curl.exe -L -o $PsExecPath -s -S https://live.sysinternals.com/PsExec64.exe
diff --git a/scripts/azure-pipelines/windows/deploy-pwsh.ps1 b/scripts/azure-pipelines/windows/deploy-pwsh.ps1 new file mode 100644 index 000000000..b766385a3 --- /dev/null +++ b/scripts/azure-pipelines/windows/deploy-pwsh.ps1 @@ -0,0 +1,9 @@ +# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: MIT
+
+# REPLACE WITH DROP-TO-ADMIN-USER-PREFIX.ps1
+
+# REPLACE WITH UTILITY-PREFIX.ps1
+
+$PwshUrl = 'https://github.com/PowerShell/PowerShell/releases/download/v7.1.3/PowerShell-7.1.3-win-x64.msi'
+InstallMSI -Url $PwshUrl -Name 'PowerShell Core'
diff --git a/scripts/azure-pipelines/windows/deploy-settings.txt b/scripts/azure-pipelines/windows/deploy-settings.txt new file mode 100644 index 000000000..a80ddce39 --- /dev/null +++ b/scripts/azure-pipelines/windows/deploy-settings.txt @@ -0,0 +1,20 @@ +$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue'
+
+Write-Host 'Disabling pagefile...'
+wmic computersystem set AutomaticManagedPagefile=False
+wmic pagefileset delete
+
+$av = Get-Command Add-MPPreference -ErrorAction SilentlyContinue
+if ($null -eq $av) {
+ Write-Host 'AntiVirus not installed, skipping exclusions.'
+} else {
+ Write-Host 'Configuring AntiVirus exclusions...'
+ Add-MpPreference -ExclusionPath C:\agent
+ Add-MPPreference -ExclusionPath D:\
+ Add-MPPreference -ExclusionProcess ninja.exe
+ Add-MPPreference -ExclusionProcess clang-cl.exe
+ Add-MPPreference -ExclusionProcess cl.exe
+ Add-MPPreference -ExclusionProcess link.exe
+ Add-MPPreference -ExclusionProcess python.exe
+}
diff --git a/scripts/azure-pipelines/windows/deploy-visual-studio.ps1 b/scripts/azure-pipelines/windows/deploy-visual-studio.ps1 new file mode 100644 index 000000000..2de2de919 --- /dev/null +++ b/scripts/azure-pipelines/windows/deploy-visual-studio.ps1 @@ -0,0 +1,86 @@ +# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: MIT
+
+# REPLACE WITH DROP-TO-ADMIN-USER-PREFIX.ps1
+
+# REPLACE WITH UTILITY-PREFIX.ps1
+
+$VisualStudioBootstrapperUrl = 'https://aka.ms/vs/16/release/vs_enterprise.exe'
+$Workloads = @(
+ 'Microsoft.VisualStudio.Workload.NativeDesktop',
+ 'Microsoft.VisualStudio.Workload.Universal',
+ 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64',
+ 'Microsoft.VisualStudio.Component.VC.Tools.ARM',
+ 'Microsoft.VisualStudio.Component.VC.Tools.ARM64',
+ 'Microsoft.VisualStudio.Component.VC.ATL',
+ 'Microsoft.VisualStudio.Component.VC.ATLMFC',
+ 'Microsoft.VisualStudio.Component.VC.v141.x86.x64.Spectre',
+ 'Microsoft.VisualStudio.Component.Windows10SDK.18362',
+ 'Microsoft.VisualStudio.Component.Windows10SDK.19041',
+ 'Microsoft.Net.Component.4.8.SDK',
+ 'Microsoft.Component.NetFX.Native',
+ 'Microsoft.VisualStudio.Component.VC.Llvm.ClangToolset',
+ 'Microsoft.VisualStudio.Component.VC.Llvm.Clang',
+ 'Microsoft.VisualStudio.Component.VC.v141.x86.x64',
+ 'Microsoft.VisualStudio.Component.VC.140'
+)
+
+<#
+.SYNOPSIS
+Install Visual Studio.
+
+.DESCRIPTION
+InstallVisualStudio takes the $Workloads array, and installs it with the
+installer that's pointed at by $BootstrapperUrl.
+
+.PARAMETER Workloads
+The set of VS workloads to install.
+
+.PARAMETER BootstrapperUrl
+The URL of the Visual Studio installer, i.e. one of vs_*.exe.
+
+.PARAMETER InstallPath
+The path to install Visual Studio at.
+
+.PARAMETER Nickname
+The nickname to give the installation.
+#>
+Function InstallVisualStudio {
+ Param(
+ [String[]]$Workloads,
+ [String]$BootstrapperUrl,
+ [String]$InstallPath = $null,
+ [String]$Nickname = $null
+ )
+
+ try {
+ Write-Host 'Downloading Visual Studio...'
+ [string]$bootstrapperExe = Get-TempFilePath -Extension 'exe'
+ curl.exe -L -o $bootstrapperExe -s -S $BootstrapperUrl
+ Write-Host 'Installing Visual Studio...'
+ $vsArgs = @('/c', $bootstrapperExe, '--quiet', '--norestart', '--wait', '--nocache')
+ foreach ($workload in $Workloads) {
+ $vsArgs += '--add'
+ $vsArgs += $workload
+ }
+
+ if (-not ([String]::IsNullOrWhiteSpace($InstallPath))) {
+ $vsArgs += '--installpath'
+ $vsArgs += $InstallPath
+ }
+
+ if (-not ([String]::IsNullOrWhiteSpace($Nickname))) {
+ $vsArgs += '--nickname'
+ $vsArgs += $Nickname
+ }
+
+ $proc = Start-Process -FilePath cmd.exe -ArgumentList $vsArgs -Wait -PassThru
+ PrintMsiExitCodeMessage $proc.ExitCode
+ }
+ catch {
+ Write-Error "Failed to install Visual Studio! $($_.Exception.Message)"
+ throw
+ }
+}
+
+InstallVisualStudio -Workloads $Workloads -BootstrapperUrl $VisualStudioBootstrapperUrl -Nickname 'Stable'
diff --git a/scripts/azure-pipelines/windows/deploy-windows-wdk.ps1 b/scripts/azure-pipelines/windows/deploy-windows-wdk.ps1 new file mode 100644 index 000000000..d145f79d7 --- /dev/null +++ b/scripts/azure-pipelines/windows/deploy-windows-wdk.ps1 @@ -0,0 +1,47 @@ +# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: MIT
+
+# REPLACE WITH DROP-TO-ADMIN-USER-PREFIX.ps1
+
+# REPLACE WITH UTILITY-PREFIX.ps1
+
+$WindowsWDKUrl = 'https://go.microsoft.com/fwlink/?linkid=2128854'
+
+<#
+.SYNOPSIS
+Installs Windows WDK version 2004
+
+.DESCRIPTION
+Downloads the Windows WDK installer located at $Url, and installs it with the
+correct flags.
+
+.PARAMETER Url
+The URL of the installer.
+#>
+Function InstallWindowsWDK {
+ Param(
+ [String]$Url
+ )
+
+ try {
+ Write-Host 'Downloading Windows WDK...'
+ [string]$installerPath = Get-TempFilePath -Extension 'exe'
+ curl.exe -L -o $installerPath -s -S $Url
+ Write-Host 'Installing Windows WDK...'
+ $proc = Start-Process -FilePath $installerPath -ArgumentList @('/features', '+', '/q') -Wait -PassThru
+ $exitCode = $proc.ExitCode
+ if ($exitCode -eq 0) {
+ Write-Host 'Installation successful!'
+ }
+ else {
+ Write-Error "Installation failed! Exited with $exitCode."
+ throw
+ }
+ }
+ catch {
+ Write-Error "Failed to install Windows WDK! $($_.Exception.Message)"
+ throw
+ }
+}
+
+InstallWindowsWDK -Url $WindowsWDKUrl
diff --git a/scripts/azure-pipelines/windows/drop-to-admin-user-prefix.ps1 b/scripts/azure-pipelines/windows/drop-to-admin-user-prefix.ps1 new file mode 100644 index 000000000..b4592eabd --- /dev/null +++ b/scripts/azure-pipelines/windows/drop-to-admin-user-prefix.ps1 @@ -0,0 +1,27 @@ +param(
+ [string]$AdminUserPassword = $null
+)
+
+$ErrorActionPreference = 'Stop'
+$ProgressPreference = 'SilentlyContinue'
+if (-Not [string]::IsNullOrEmpty($AdminUserPassword)) {
+ $PsExecPath = 'C:\PsExec64.exe'
+ $PsExecArgs = @(
+ '-u',
+ 'AdminUser',
+ '-p',
+ $AdminUserPassword,
+ '-accepteula',
+ '-i',
+ '-h',
+ 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe',
+ '-ExecutionPolicy',
+ 'Unrestricted',
+ '-File',
+ $PSCommandPath
+ )
+
+ Write-Host "Executing: $PsExecPath $PsExecArgs"
+ $proc = Start-Process -FilePath $PsExecPath -ArgumentList $PsExecArgs -Wait -PassThru
+ exit $proc.ExitCode
+}
diff --git a/scripts/azure-pipelines/windows/provision-entire-image.ps1 b/scripts/azure-pipelines/windows/provision-entire-image.ps1 new file mode 100644 index 000000000..3bfb5dbd9 --- /dev/null +++ b/scripts/azure-pipelines/windows/provision-entire-image.ps1 @@ -0,0 +1,16 @@ +# This script runs all the scripts we run on Azure machines to deploy prerequisites,
+# and assumes it is being run as an admin user.
+
+. "$PSScriptRoot\utility-prefix.ps1"
+
+. "$PSScriptRoot\deploy-visual-studio.ps1"
+. "$PSScriptRoot\deploy-windows-wdk.ps1"
+. "$PSScriptRoot\deploy-mpi.ps1"
+. "$PSScriptRoot\deploy-cuda.ps1"
+. "$PSScriptRoot\deploy-pwsh.ps1"
+try {
+ Copy-Item "$PSScriptRoot\deploy-settings.txt" "$PSScriptRoot\deploy-settings.ps1"
+ . "$PSScriptRoot\deploy-settings.ps1"
+} finally {
+ Remove-Item "$PSScriptRoot\deploy-settings.ps1"
+}
diff --git a/scripts/azure-pipelines/windows/provision-image.txt b/scripts/azure-pipelines/windows/provision-image.txt deleted file mode 100644 index e3e04be28..000000000 --- a/scripts/azure-pipelines/windows/provision-image.txt +++ /dev/null @@ -1,454 +0,0 @@ -# Copyright (c) Microsoft Corporation.
-# SPDX-License-Identifier: MIT
-
-<#
-.SYNOPSIS
-Sets up a machine to be an image for a scale set.
-
-.DESCRIPTION
-provision-image.ps1 runs on an existing, freshly provisioned virtual machine,
-and sets up that virtual machine as a build machine. After this is done,
-(outside of this script), we take that machine and make it an image to be copied
-for setting up new VMs in the scale set.
-
-This script must either be run as admin, or one must pass AdminUserPassword;
-if the script is run with AdminUserPassword, it runs itself again as an
-administrator.
-
-.PARAMETER AdminUserPassword
-The administrator user's password; if this is $null, or not passed, then the
-script assumes it's running on an administrator account.
-
-.PARAMETER StorageAccountName
-The name of the storage account. Stored in the environment variable %PROVISIONED_AZURE_STORAGE_NAME%.
-Used by the CI system to access the global storage.
-
-.PARAMETER StorageAccountSasToken
-The SAS token to access the storage account. Stored in the environment variable
-%PROVISIONED_AZURE_STORAGE_SAS_TOKEN%.
-Used by the CI system to access the global storage.
-#>
-param(
- [string]$AdminUserPassword = $null,
- [string]$StorageAccountName = $null,
- [string]$StorageAccountSasToken = $null
-)
-
-$ErrorActionPreference = 'Stop'
-
-<#
-.SYNOPSIS
-Gets a random file path in the temp directory.
-
-.DESCRIPTION
-Get-TempFilePath takes an extension, and returns a path with a random
-filename component in the temporary directory with that extension.
-
-.PARAMETER Extension
-The extension to use for the path.
-#>
-Function Get-TempFilePath {
- Param(
- [String]$Extension
- )
-
- if ([String]::IsNullOrWhiteSpace($Extension)) {
- throw 'Missing Extension'
- }
-
- $tempPath = [System.IO.Path]::GetTempPath()
- $tempName = [System.IO.Path]::GetRandomFileName() + '.' + $Extension
- return Join-Path $tempPath $tempName
-}
-
-$TranscriptPath = 'C:\provision-image-transcript.txt'
-
-if ([string]::IsNullOrEmpty($AdminUserPassword)) {
- Start-Transcript -Path $TranscriptPath
-} else {
- Write-Host 'AdminUser password supplied; switching to AdminUser.'
- $PsExecPath = Get-TempFilePath -Extension 'exe'
- Write-Host "Downloading psexec to: $PsExecPath"
- & curl.exe -L -o $PsExecPath -s -S https://live.sysinternals.com/PsExec64.exe
- $PsExecArgs = @(
- '-u',
- 'AdminUser',
- '-p',
- $AdminUserPassword,
- '-accepteula',
- '-h',
- 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe',
- '-ExecutionPolicy',
- 'Unrestricted',
- '-File',
- $PSCommandPath
- )
-
- if (-Not ([string]::IsNullOrWhiteSpace($StorageAccountName))) {
- $PsExecArgs += '-StorageAccountName'
- $PsExecArgs += $StorageAccountName
- }
-
- if (-Not ([string]::IsNullOrWhiteSpace($StorageAccountSasToken))) {
- $PsExecArgs += '-StorageAccountSasToken'
- $PsExecArgs += $StorageAccountSasToken
- }
-
- Write-Host "Executing: $PsExecPath $PsExecArgs"
-
- $proc = Start-Process -FilePath $PsExecPath -ArgumentList $PsExecArgs -Wait -PassThru
- Write-Host 'Reading transcript...'
- Get-Content -Path $TranscriptPath
- Write-Host 'Cleaning up...'
- Remove-Item $PsExecPath
- exit $proc.ExitCode
-}
-
-$VisualStudioBootstrapperUrl = 'https://aka.ms/vs/16/release/vs_enterprise.exe'
-$Workloads = @(
- 'Microsoft.VisualStudio.Workload.NativeDesktop',
- 'Microsoft.VisualStudio.Workload.Universal',
- 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64',
- 'Microsoft.VisualStudio.Component.VC.Tools.ARM',
- 'Microsoft.VisualStudio.Component.VC.Tools.ARM64',
- 'Microsoft.VisualStudio.Component.VC.ATL',
- 'Microsoft.VisualStudio.Component.VC.ATLMFC',
- 'Microsoft.VisualStudio.Component.VC.v141.x86.x64.Spectre',
- 'Microsoft.VisualStudio.Component.Windows10SDK.18362',
- 'Microsoft.VisualStudio.Component.Windows10SDK.19041',
- 'Microsoft.Net.Component.4.8.SDK',
- 'Microsoft.Component.NetFX.Native',
- 'Microsoft.VisualStudio.Component.VC.Llvm.ClangToolset',
- 'Microsoft.VisualStudio.Component.VC.Llvm.Clang',
- 'Microsoft.VisualStudio.Component.VC.v141.x86.x64',
- 'Microsoft.VisualStudio.Component.VC.140'
-)
-
-$WindowsWDKUrl = 'https://go.microsoft.com/fwlink/?linkid=2128854'
-
-$MpiUrl = 'https://download.microsoft.com/download/a/5/2/a5207ca5-1203-491a-8fb8-906fd68ae623/msmpisetup.exe'
-
-$CudaUrl = 'https://developer.download.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda_10.1.243_426.00_win10.exe'
-$CudaFeatures = 'nvcc_10.1 cuobjdump_10.1 nvprune_10.1 cupti_10.1 gpu_library_advisor_10.1 memcheck_10.1 ' + `
- 'nvdisasm_10.1 nvprof_10.1 visual_profiler_10.1 visual_studio_integration_10.1 cublas_10.1 cublas_dev_10.1 ' + `
- 'cudart_10.1 cufft_10.1 cufft_dev_10.1 curand_10.1 curand_dev_10.1 cusolver_10.1 cusolver_dev_10.1 cusparse_10.1 ' + `
- 'cusparse_dev_10.1 nvgraph_10.1 nvgraph_dev_10.1 npp_10.1 npp_dev_10.1 nvrtc_10.1 nvrtc_dev_10.1 nvml_dev_10.1 ' + `
- 'occupancy_calculator_10.1 fortran_examples_10.1'
-
-$PwshUrl = 'https://github.com/PowerShell/PowerShell/releases/download/v7.1.0/PowerShell-7.1.0-win-x64.msi'
-
-$ErrorActionPreference = 'Stop'
-$ProgressPreference = 'SilentlyContinue'
-
-<#
-.SYNOPSIS
-Writes a message to the screen depending on ExitCode.
-
-.DESCRIPTION
-Since msiexec can return either 0 or 3010 successfully, in both cases
-we write that installation succeeded, and which exit code it exited with.
-If msiexec returns anything else, we write an error.
-
-.PARAMETER ExitCode
-The exit code that msiexec returned.
-#>
-Function PrintMsiExitCodeMessage {
- Param(
- $ExitCode
- )
-
- # 3010 is probably ERROR_SUCCESS_REBOOT_REQUIRED
- if ($ExitCode -eq 0 -or $ExitCode -eq 3010) {
- Write-Host "Installation successful! Exited with $ExitCode."
- }
- else {
- Write-Error "Installation failed! Exited with $ExitCode."
- throw
- }
-}
-
-<#
-.SYNOPSIS
-Install Visual Studio.
-
-.DESCRIPTION
-InstallVisualStudio takes the $Workloads array, and installs it with the
-installer that's pointed at by $BootstrapperUrl.
-
-.PARAMETER Workloads
-The set of VS workloads to install.
-
-.PARAMETER BootstrapperUrl
-The URL of the Visual Studio installer, i.e. one of vs_*.exe.
-
-.PARAMETER InstallPath
-The path to install Visual Studio at.
-
-.PARAMETER Nickname
-The nickname to give the installation.
-#>
-Function InstallVisualStudio {
- Param(
- [String[]]$Workloads,
- [String]$BootstrapperUrl,
- [String]$InstallPath = $null,
- [String]$Nickname = $null
- )
-
- try {
- Write-Host 'Downloading Visual Studio...'
- [string]$bootstrapperExe = Get-TempFilePath -Extension 'exe'
- curl.exe -L -o $bootstrapperExe -s -S $BootstrapperUrl
- Write-Host 'Installing Visual Studio...'
- $args = @('/c', $bootstrapperExe, '--quiet', '--norestart', '--wait', '--nocache')
- foreach ($workload in $Workloads) {
- $args += '--add'
- $args += $workload
- }
-
- if (-not ([String]::IsNullOrWhiteSpace($InstallPath))) {
- $args += '--installpath'
- $args += $InstallPath
- }
-
- if (-not ([String]::IsNullOrWhiteSpace($Nickname))) {
- $args += '--nickname'
- $args += $Nickname
- }
-
- $proc = Start-Process -FilePath cmd.exe -ArgumentList $args -Wait -PassThru
- PrintMsiExitCodeMessage $proc.ExitCode
- }
- catch {
- Write-Error "Failed to install Visual Studio! $($_.Exception.Message)"
- throw
- }
-}
-
-<#
-.SYNOPSIS
-Install a .msi file.
-
-.DESCRIPTION
-InstallMSI takes a url where an .msi lives, and installs that .msi to the system.
-
-.PARAMETER Name
-The name of the thing to install.
-
-.PARAMETER Url
-The URL at which the .msi lives.
-#>
-Function InstallMSI {
- Param(
- [String]$Name,
- [String]$Url
- )
-
- try {
- Write-Host "Downloading $Name..."
- [string]$msiPath = Get-TempFilePath -Extension 'msi'
- curl.exe -L -o $msiPath -s -S $Url
- Write-Host "Installing $Name..."
- $args = @('/i', $msiPath, '/norestart', '/quiet', '/qn')
- $proc = Start-Process -FilePath 'msiexec.exe' -ArgumentList $args -Wait -PassThru
- PrintMsiExitCodeMessage $proc.ExitCode
- }
- catch {
- Write-Error "Failed to install $Name! $($_.Exception.Message)"
- throw
- }
-}
-
-<#
-.SYNOPSIS
-Unpacks a zip file to $Dir.
-
-.DESCRIPTION
-InstallZip takes a URL of a zip file, and unpacks the zip file to the directory
-$Dir.
-
-.PARAMETER Name
-The name of the tool being installed.
-
-.PARAMETER Url
-The URL of the zip file to unpack.
-
-.PARAMETER Dir
-The directory to unpack the zip file to.
-#>
-Function InstallZip {
- Param(
- [String]$Name,
- [String]$Url,
- [String]$Dir
- )
-
- try {
- Write-Host "Downloading $Name..."
- [string]$zipPath = Get-TempFilePath -Extension 'zip'
- curl.exe -L -o $zipPath -s -S $Url
- Write-Host "Installing $Name..."
- Expand-Archive -Path $zipPath -DestinationPath $Dir -Force
- }
- catch {
- Write-Error "Failed to install $Name! $($_.Exception.Message)"
- throw
- }
-}
-
-<#
-.SYNOPSIS
-Installs Windows WDK version 2004
-
-.DESCRIPTION
-Downloads the Windows WDK installer located at $Url, and installs it with the
-correct flags.
-
-.PARAMETER Url
-The URL of the installer.
-#>
-Function InstallWindowsWDK {
- Param(
- [String]$Url
- )
-
- try {
- Write-Host 'Downloading Windows WDK...'
- [string]$installerPath = Get-TempFilePath -Extension 'exe'
- curl.exe -L -o $installerPath -s -S $Url
- Write-Host 'Installing Windows WDK...'
- $proc = Start-Process -FilePath $installerPath -ArgumentList @('/features', '+', '/q') -Wait -PassThru
- $exitCode = $proc.ExitCode
- if ($exitCode -eq 0) {
- Write-Host 'Installation successful!'
- }
- else {
- Write-Error "Installation failed! Exited with $exitCode."
- throw
- }
- }
- catch {
- Write-Error "Failed to install Windows WDK! $($_.Exception.Message)"
- throw
- }
-}
-
-<#
-.SYNOPSIS
-Installs MPI
-
-.DESCRIPTION
-Downloads the MPI installer located at $Url, and installs it with the
-correct flags.
-
-.PARAMETER Url
-The URL of the installer.
-#>
-Function InstallMpi {
- Param(
- [String]$Url
- )
-
- try {
- Write-Host 'Downloading MPI...'
- [string]$installerPath = Get-TempFilePath -Extension 'exe'
- curl.exe -L -o $installerPath -s -S $Url
- Write-Host 'Installing MPI...'
- $proc = Start-Process -FilePath $installerPath -ArgumentList @('-force', '-unattend') -Wait -PassThru
- $exitCode = $proc.ExitCode
- if ($exitCode -eq 0) {
- Write-Host 'Installation successful!'
- }
- else {
- Write-Error "Installation failed! Exited with $exitCode."
- throw
- }
- }
- catch {
- Write-Error "Failed to install MPI! $($_.Exception.Message)"
- throw
- }
-}
-
-<#
-.SYNOPSIS
-Installs NVIDIA's CUDA Toolkit.
-
-.DESCRIPTION
-InstallCuda installs the CUDA Toolkit with the features specified as a
-space-separated list of strings in $Features.
-
-.PARAMETER Url
-The URL of the CUDA installer.
-
-.PARAMETER Features
-A space-separated list of features to install.
-#>
-Function InstallCuda {
- Param(
- [String]$Url,
- [String]$Features
- )
-
- try {
- Write-Host 'Downloading CUDA...'
- [string]$installerPath = Get-TempFilePath -Extension 'exe'
- curl.exe -L -o $installerPath -s -S $Url
- Write-Host 'Installing CUDA...'
- $proc = Start-Process -FilePath $installerPath -ArgumentList @('-s ' + $Features) -Wait -PassThru
- $exitCode = $proc.ExitCode
- if ($exitCode -eq 0) {
- Write-Host 'Installation successful!'
- }
- else {
- Write-Error "Installation failed! Exited with $exitCode."
- throw
- }
- }
- catch {
- Write-Error "Failed to install CUDA! $($_.Exception.Message)"
- throw
- }
-}
-
-Write-Host "AdminUser password not supplied; assuming already running as AdminUser"
-
-Write-Host 'Disabling pagefile...'
-wmic computersystem set AutomaticManagedPagefile=False
-wmic pagefileset delete
-
-$av = Get-Command Add-MPPreference -ErrorAction SilentlyContinue
-if ($null -eq $av) {
- Write-Host 'AntiVirus not installed, skipping exclusions.'
-} else {
- Write-Host 'Configuring AntiVirus exclusions...'
- Add-MpPreference -ExclusionPath C:\agent
- Add-MPPreference -ExclusionPath D:\
- Add-MPPreference -ExclusionProcess ninja.exe
- Add-MPPreference -ExclusionProcess clang-cl.exe
- Add-MPPreference -ExclusionProcess cl.exe
- Add-MPPreference -ExclusionProcess link.exe
- Add-MPPreference -ExclusionProcess python.exe
-}
-
-InstallVisualStudio -Workloads $Workloads -BootstrapperUrl $VisualStudioBootstrapperUrl -Nickname 'Stable'
-InstallWindowsWDK -Url $WindowsWDKUrl
-InstallMpi -Url $MpiUrl
-InstallCuda -Url $CudaUrl -Features $CudaFeatures
-InstallMSI -Url $PwshUrl -Name 'PowerShell Core'
-if ([string]::IsNullOrWhiteSpace($StorageAccountName)) {
- Write-Host 'No storage account name configured.'
-} else {
- Write-Host 'Storing storage account name to environment'
- Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\Environment' `
- -Name PROVISIONED_AZURE_STORAGE_NAME `
- -Value $StorageAccountName
-}
-if ([string]::IsNullOrWhiteSpace($StorageAccountSasToken)) {
- Write-Host 'No storage account key configured.'
-} else {
- Write-Host 'Storing storage account key to environment'
- Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\Environment' `
- -Name PROVISIONED_AZURE_STORAGE_SAS_TOKEN `
- -Value $StorageAccountSasToken
-}
diff --git a/scripts/azure-pipelines/windows/utility-prefix.ps1 b/scripts/azure-pipelines/windows/utility-prefix.ps1 new file mode 100644 index 000000000..8cd0066db --- /dev/null +++ b/scripts/azure-pipelines/windows/utility-prefix.ps1 @@ -0,0 +1,125 @@ +# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: MIT
+
+<#
+.SYNOPSIS
+Gets a random file path in the temp directory.
+
+.DESCRIPTION
+Get-TempFilePath takes an extension, and returns a path with a random
+filename component in the temporary directory with that extension.
+
+.PARAMETER Extension
+The extension to use for the path.
+#>
+Function Get-TempFilePath {
+ Param(
+ [String]$Extension
+ )
+
+ if ([String]::IsNullOrWhiteSpace($Extension)) {
+ throw 'Missing Extension'
+ }
+
+ $tempPath = [System.IO.Path]::GetTempPath()
+ $tempName = [System.IO.Path]::GetRandomFileName() + '.' + $Extension
+ return Join-Path $tempPath $tempName
+}
+
+<#
+.SYNOPSIS
+Writes a message to the screen depending on ExitCode.
+
+.DESCRIPTION
+Since msiexec can return either 0 or 3010 successfully, in both cases
+we write that installation succeeded, and which exit code it exited with.
+If msiexec returns anything else, we write an error.
+
+.PARAMETER ExitCode
+The exit code that msiexec returned.
+#>
+Function PrintMsiExitCodeMessage {
+ Param(
+ $ExitCode
+ )
+
+ # 3010 is probably ERROR_SUCCESS_REBOOT_REQUIRED
+ if ($ExitCode -eq 0 -or $ExitCode -eq 3010) {
+ Write-Host "Installation successful! Exited with $ExitCode."
+ }
+ else {
+ Write-Error "Installation failed! Exited with $ExitCode."
+ throw
+ }
+}
+
+<#
+.SYNOPSIS
+Install a .msi file.
+
+.DESCRIPTION
+InstallMSI takes a url where an .msi lives, and installs that .msi to the system.
+
+.PARAMETER Name
+The name of the thing to install.
+
+.PARAMETER Url
+The URL at which the .msi lives.
+#>
+Function InstallMSI {
+ Param(
+ [String]$Name,
+ [String]$Url
+ )
+
+ try {
+ Write-Host "Downloading $Name..."
+ [string]$msiPath = Get-TempFilePath -Extension 'msi'
+ curl.exe -L -o $msiPath -s -S $Url
+ Write-Host "Installing $Name..."
+ $args = @('/i', $msiPath, '/norestart', '/quiet', '/qn')
+ $proc = Start-Process -FilePath 'msiexec.exe' -ArgumentList $args -Wait -PassThru
+ PrintMsiExitCodeMessage $proc.ExitCode
+ }
+ catch {
+ Write-Error "Failed to install $Name! $($_.Exception.Message)"
+ throw
+ }
+}
+
+<#
+.SYNOPSIS
+Unpacks a zip file to $Dir.
+
+.DESCRIPTION
+InstallZip takes a URL of a zip file, and unpacks the zip file to the directory
+$Dir.
+
+.PARAMETER Name
+The name of the tool being installed.
+
+.PARAMETER Url
+The URL of the zip file to unpack.
+
+.PARAMETER Dir
+The directory to unpack the zip file to.
+#>
+Function InstallZip {
+ Param(
+ [String]$Name,
+ [String]$Url,
+ [String]$Dir
+ )
+
+ try {
+ Write-Host "Downloading $Name..."
+ [string]$zipPath = Get-TempFilePath -Extension 'zip'
+ curl.exe -L -o $zipPath -s -S $Url
+ Write-Host "Installing $Name..."
+ Expand-Archive -Path $zipPath -DestinationPath $Dir -Force
+ }
+ catch {
+ Write-Error "Failed to install $Name! $($_.Exception.Message)"
+ throw
+ }
+}
diff --git a/scripts/ci.baseline.txt b/scripts/ci.baseline.txt index 01b33f2d1..9ced1bd96 100644 --- a/scripts/ci.baseline.txt +++ b/scripts/ci.baseline.txt @@ -1784,7 +1784,6 @@ clapack:x64-uwp=skip lapack-reference:arm64-windows=skip lapack-reference:arm-uwp=skip - # failures for x64-windows-static-md ace:x64-windows-static-md=fail activemq-cpp:x64-windows-static-md=fail @@ -1819,3 +1818,15 @@ zyre:x64-windows-static-md=fail usbmuxd:x64-windows-static-md=fail workflow:x64-uwp=fail workflow:arm-uwp=fail + +# wangle triggers an internal compiler error +# https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_workitems/edit/1269468 +wangle:x64-windows=fail +wangle:x64-windows-static=fail +wangle:x64-windows-static-md=fail + +# VS2019 version 16.9.4's project system changes where PDBs are placed in a way that breaks the +# upstream build script of this port. +# See https://developercommunity.visualstudio.com/t/Toolset-169-regression-vcxproj-producin/1356639 +dimcli:x64-windows-static-md=fail +dimcli:x64-windows-static=fail diff --git a/scripts/vcpkgTools.xml b/scripts/vcpkgTools.xml index 50e0476fe..3d48d5294 100644 --- a/scripts/vcpkgTools.xml +++ b/scripts/vcpkgTools.xml @@ -8,25 +8,25 @@ <archiveName>python-3.9.2-embed-win32.zip</archiveName> </tool> <tool name="cmake" os="windows"> - <version>3.19.2</version> - <exeRelativePath>cmake-3.19.2-win32-x86\bin\cmake.exe</exeRelativePath> - <url>https://github.com/Kitware/CMake/releases/download/v3.19.2/cmake-3.19.2-win32-x86.zip</url> - <sha512>1ed3706d8dd5c5244ceb4c22443f006fcb9ffa76cb6d9ebdbbb8f11a50deb3f92a9a4080ff3fd125f79c0143f553ea0c02a6a2d27521438705c2fea4dd97d239</sha512> - <archiveName>cmake-3.19.2-win32-x86.zip</archiveName> + <version>3.20.1</version> + <exeRelativePath>cmake-3.20.1-windows-i386\bin\cmake.exe</exeRelativePath> + <url>https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1-windows-i386.zip</url> + <sha512>bc0be4e32bbffe102b2cb970591118bf2132d6afd6c97469cf46a4f772627fa81a8a57e4069dc32a93c0ffcce176e562d9d008e6780f053876e5d5eb97051554</sha512> + <archiveName>cmake-3.20.1-windows-i386.zip</archiveName> </tool> <tool name="cmake" os="osx"> - <version>3.19.2</version> - <exeRelativePath>cmake-3.19.2-macos-universal/CMake.app/Contents/bin/cmake</exeRelativePath> - <url>https://github.com/Kitware/CMake/releases/download/v3.19.2/cmake-3.19.2-macos-universal.tar.gz</url> - <sha512>031196439a26d6ff2a1f42830ca26b5159752dc2c2c889a4e304327c858157b68088905b10727090918c3605fe7fe429aa19d46e68ed30c17d4d89b3d209aae8</sha512> - <archiveName>cmake-3.19.2-macos-universal.tar.gz</archiveName> + <version>3.20.1</version> + <exeRelativePath>cmake-3.20.1-macos-universal/CMake.app/Contents/bin/cmake</exeRelativePath> + <url>https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1-macos-universal.tar.gz</url> + <sha512>fd17d09a0639833a2aa5f30045cfcbbb30e0cfa5092d2f22f0bc502ea361a458d26be96437fbfb6b5e68b73954050c0256653af7e2e20bb253a878e206992135</sha512> + <archiveName>cmake-3.20.1-macos-universal.tar.gz</archiveName> </tool> <tool name="cmake" os="linux"> - <version>3.19.2</version> - <exeRelativePath>cmake-3.19.2-Linux-x86_64/bin/cmake</exeRelativePath> - <url>https://github.com/Kitware/CMake/releases/download/v3.19.2/cmake-3.19.2-Linux-x86_64.tar.gz</url> - <sha512>67ea7ee428e6c93039646d6d0cbe84c0be337a2b878487635fee71ec2c8992a9078a660e6b793ef11ce13530def44a16b716af96e10a97675d014a26caa0aed4</sha512> - <archiveName>cmake-3.19.2-Linux-x86_64.tar.gz</archiveName> + <version>3.20.1</version> + <exeRelativePath>cmake-3.20.1-linux-x86_64/bin/cmake</exeRelativePath> + <url>https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1-Linux-x86_64.tar.gz</url> + <sha512>e14b6cbc03abfbae074c36e34f90eea94af02e727e75ed597f748203e2ce7483ffe693bd173cb02a1de7ac55254a891da5505c43b92528adc918eff9e8b024e6</sha512> + <archiveName>cmake-3.20.1-linux-x86_64.tar.gz</archiveName> </tool> <tool name="cmake" os="freebsd"> <version>3.12.4</version> @@ -134,10 +134,10 @@ <archiveName>ninja-freebsd-1.8.2.zip</archiveName> </tool> <tool name="powershell-core" os="windows"> - <version>7.1.0</version> + <version>7.1.3</version> <exeRelativePath>pwsh.exe</exeRelativePath> - <url>https://github.com/PowerShell/PowerShell/releases/download/v7.1.0/PowerShell-7.1.0-win-x86.zip</url> - <sha512>ea6ed619c784af65b09a5e98cdfc0d41333ba7373b7409e488ec70aaf1e0c4e5568e846d2b1748d72c3e3a9c1a4827f0ba29e9f6e93e79b4f49312ca40af6d2a</sha512> - <archiveName>PowerShell-7.1.0-win-x86.zip</archiveName> + <url>https://github.com/PowerShell/PowerShell/releases/download/v7.1.3/PowerShell-7.1.3-win-x86.zip</url> + <sha512>8c2ce510b5c641aad2da6adefc92d47e09bc842d47db3b5d15e14859555a74fe13ad52eaeabf1b2954ca9af737e628b567731c8a3db9bbf0e4aad05279bc1fd8</sha512> + <archiveName>PowerShell-7.1.3-win-x86.zip</archiveName> </tool> </tools> |