Rewiring

Azure Pipelines in action pt. III - reasonably secure deploys to IIS

Update: the standard way of doing IIS deployments is hosting the agent on the app server and running tools such as IisWebAppDeployment.

This post describes an approach where there is one centralized agent, and the deployments to app servers are done via Powershell remoting

I'll return to the tradeoffs later, when I've had a chance to implement and play with the first one. As to why we didn't go with the default: long story short - we inherited the original idea and ran with it.

Trying to implement a deployment flow from Azure Devops to an on-prem IIS server, where every account is not an administrator, is a surprisingly difficult task. Here's an approach that works for us, and might for you too!

My other Azure posts

One of the major wins on our automation journey thus far has been automating deployments to our internal IIS servers. This includes things like permission minimization, application pool manipulation and a huge number of gotchas. Here I'll try to go into as much detail as I can, but have to note from the get go: I'm not a Windows administrator, have little knowledge about networking and fledgling skills in PowerShell. Regardless, I feel like we've achieved something that is worth sharing.

The requirements

The template

The goal is to achieve a pipeline template that:

The template we're going to work with is a simple one with as little bells and whistles we can have to get this thing working. To start with, we expect to have an artifact ready to go.

See the whole thing if you want to absorb it all first before analyzing piece by piece.

Let's dig in!

The deployment job

We're running everything in a single job. This is important to guarantee that the whole operation is ran on the same agent instance.

jobs:
  - deployment:
    ${{ if not(eq(length(parameters.dependsOn), 0))}}:
      dependsOn: ${{ parameters.dependsOn }}
    workspace:
      clean: all # always clean workspace on self-hosted agents
    displayName: Update application on on-prem IIS (${{ parameters.deploymentEnvironmentName }})
    environment: ${{ parameters.deploymentEnvironmentName }}
    ${{ if parameters.pool }}:
      pool: ${{parameters.pool}}
    variables:
      - ${{ each g in parameters.deploymentVariableGroups }}: # take in one or more variable group containing the deployment variables
          - group: ${{g}}
      - name: zipFileOnBuildMachine
        value: $(Pipeline.Workspace)/${{ parameters.artifactName }}
      - name: artifactZipPath
        value: $(Pipeline.Workspace)/${{ parameters.artifactName }}/**/*.zip

    strategy:
      runOnce:
        deploy:
          steps:

The backbone for running pwsh on the target machine

Here's a helper template for running PowerShell on the target machines. The original PowershellOnTargetMachines requires an administrator account (when it seems this could be changed in a single line), which we do NOT want. As a workaround we must write quite a lot of PowerShell, which of course beggars the question of why we're even using YAML in the first place...

parameters:
  - name: username
    type: string

  - name: password
    type: string

  # The name of the JEA config which is providing the installation user the proper access
  - name: jeaConfig
    type: string
    default: ''

  # Name of the target server
  - name: machineName
    type: string

  # the script block to run
  - name: scriptBlock
    type: string

  - name: displayName
    type: string

  - name: script
    type: string
    default: ''

steps:
  - pwsh: |
      $securePassword = ConvertTo-SecureString -AsPlainText '${{ parameters.password }}' -Force
      $credential = New-Object System.Management.Automation.PSCredential('${{ parameters.username }}', $securePassword)
      $sessionParams = @{
        ComputerName = '${{ parameters.machineName }}'
        Credential   = $credential
      }
      if ('${{ parameters.jeaConfig }}' -ne '') {
        $sessionParams.ConfigurationName = '${{ parameters.jeaConfig }}'
      }
      $session = New-PSSession @sessionParams

      $start = @{
        Session      = $session
        ScriptBlock  = {
          ${{ parameters.scriptBlock }}
        }
      }

      Invoke-Command @start
      ${{ parameters.script }}

      $session | Remove-PsSession
    displayName: ${{ parameters.displayName }}

Now that the base is set up, let's walk through each step in this deployment:

Move the artifact

Here we first download the artifact to the agent machine and move it over to the target machine.

            - checkout: none
            - download: none
            - task: DownloadPipelineArtifact@2
              displayName: Download build artifact
              inputs:
                source: current
                runVersion: latest
                artifactName: ${{ parameters.artifactName }}
                targetPath: $(zipFileOnBuildMachine)

            - template: /steps/powershell-on-target-machine.yml
              parameters:
                username: $(username)
                password: $(userPassword)
                machineName: $(machineName)
                scriptBlock: |
                  if (Test-Path -Path "${{parameters.artifactDirectory}}") {
                    Remove-Item -Path "${{parameters.artifactDirectory}}" -Recurse -Force
                  }
                script: Copy-Item $(zipFileOnBuildMachine) -Destination "${{parameters.artifactDirectory}}" -ToSession $session -Recurse
                displayName: Transfer artifact to target machine

First snag: If you created a fresh domain account for this, it has no way to access the target machine. Let's fix that first with this Powershell script:

$user = <your domain user>
$directories = <the artifact and app directories the user needs access to>

$remoteMgtGroupName = "Remote Management Users"
Write-Output "Adding user to ${remoteMgtGroupName} group"
try {
    net localgroup $remoteMgtGroupName $user /add
}
catch [System.Management.Automation.RemoteException] {
    if ($_ -like "System error 1378 has occurred.") {
        "User ${user} is already a member of ${remoteMgtGroupName} group."
    }
    else {
        throw
    }
}

foreach ($directory in $directories) {
  if (-Not (Test-Path -Path $directory)) {
      New-Item -ItemType Directory -Path $directory
      Write-Output "Directory '$directory' created."
  }
  
  $acl = Get-Acl $directory
  $accessRule = New-Object System.Security.AccessControl.FileSystemAccessRule($user, "Modify", "ContainerInherit,ObjectInherit", "None", "Allow")
  $acl.SetAccessRule($accessRule)
  Set-Acl -Path $directory -AclObject $acl
}

Remote Management Users seems overkill for this, but WinRMRemoteWMIUsers_ has been kinda deprecated in newer versions so I don't know if that should be dependent upon. Please let me know if you have better options here!

After this, the artifact should be available in the target machine in the directory you defined in the artifactDirectory parameter.

Shut down the application pool

Here we use the IISAdministration module to manipulate the application pool.

Since we're using JEA, we can't just directly call Get-IISAppPool in yaml, since the language mode is limited. Instead, we create a shut-down-iis-app-pool.ps1 script on the target machine:

[CmdletBinding()]
param(
    [Parameter(Mandatory = $true)]
    [string]$PoolName
)

$appPool = Get-IISAppPool -Name $PoolName;
if ($appPool.State -eq "Stopped")
{
    Write-Output "Application pool $AppPoolName is already stopped."
    return;
}

Write-Output "Stopping Application pool $AppPoolName"
try
{
    $appPool.Stop()
}
catch
{
    Write-Output $appPool
    Throw "Failed to stop Application pool $AppPoolName. Error: $_"
}

# app pool shutdown occasionally takes a while
$timeout = 20  # seconds
$elapsed = 0
while ($appPool.State -eq "Stopping" -and $elapsed -lt $timeout)
{
    Start-Sleep -Seconds 2
    $elapsed += 2
}
if ($appPool.State -ne "Stopped")
{
    Write-Output $appPool.State
    Throw "Application pool $AppPoolName could not be stopped within the timeout period."
}

Once this in in place, we can call it from the pipeline:

- template: /steps/powershell-on-target-machine.yml
  parameters:
	username: $(username)
	password: $(userPassword)
	jeaConfig: ${{parameters.jeaConfig}}
	machineName: $(machineName)
	scriptBlock: '& C:\scripts\shut-down-iis-app-pool.ps1 -PoolName $(appPoolName)'
	displayName: Stop application pool

But of course, we haven't yet allowed access to the script file. We want to limit the capabilities of the installation account to the bare minimum, so we only allow it to manipulate this specific application pool. Let's see about the JEA config at this point.

We have a .ps1 script for managing these, but I'm hesitant to share it at this point since it's pretty fickle and I'm sure there's bugs buried in there somewhere. Email me if you want a peek!

Instead of the management script, here's a JEA config that allows the user to call the specific script files and only for a specific application pool:

RoleCapability file created with New-PsRoleCapabilityFile:

@{

# ID used to uniquely identify this document
GUID = '9c85f746-8b9e-4866-95b7-20e19ac9dc94'

# Description of the functionality provided by these settings
Description = 'Allows specific management of IIS Application Pools'

# Modules to import when applied to a session
ModulesToImport = 'IISAdministration'

# Cmdlets to make visible when applied to a session
VisibleCmdlets = @{
    'Name' = 'Get-IISAppPool'
    'Parameters' = @{
        'Name' = 'Name'
        'ValidateSet' = 'app-pool-name' } }

# External commands (scripts and applications) to make visible when applied to a session
VisibleExternalCommands = 'C:\scripts\shut-down-iis-app-pool.ps1', 'C:\scripts\start-IIS-app-pool.ps1'

# Providers to make visible when applied to a session
VisibleProviders = 'IISAdministration'
}

SessionConfiguration file created with New-PSSessionConfigurationFile:

@{

# Version number of the schema used for this document
SchemaVersion = '2.0.0.0'

# ID used to uniquely identify this document
GUID = 'b9f59eb5-904a-4747-b777-0414af95b2f8'

# Description of the functionality provided by these settings
Description = 'JEA Role for managing IIS Application Pool'

# Session type defaults to apply for this session configuration. Can be 'RestrictedRemoteServer' (recommended), 'Empty', or 'Default'
SessionType = 'RestrictedRemoteServer'

# Directory to place session transcripts for this session configuration
TranscriptDirectory = 'C:\Transcripts\'

# Whether to run this session configuration as the machine's (virtual) administrator account
RunAsVirtualAccount = $true

# User roles (security groups), and the role capabilities that should be applied to them when applied to a session
RoleDefinitions = @{
    'yourUser' = @{
        'RoleCapabilityFiles' = 'path\to\file.psrc' } }
}

As far as I can tell, RunAsVirtualAccount is required here, although it immediately makes the account a "virtual administrator". I guess you could configure the groups the virtual account runs as, but seeing as the default is local Administrators, and this is the recommended approach, it's all good 🤷. I would imagine that "Just Enough Administration" would start at "no permissions at all", but apparently we go with "full access by default". I swear, PowerShell remoting...

After you've registered the JEA config, the pipeline should be allowed to pass and the application pool should be shut down.

Overwriting the application directory

Once the app pool is stopped, we overwrite the files in the app directory with the new ones from the artifact. If need be, we could keep e.g. web.config or appsettings.Production.json here, but here we assume your artifact has everything the app needs.

- template: /steps/powershell-on-target-machine.yml
  parameters:
	username: $(username)
	password: $(userPassword)
	machineName: $(machineName)
	scriptBlock: |
	  $TargetPath = "${{parameters.appDirectory}}"
	  $SourcePath = "${{parameters.artifactDirectory}}"
	  Write-Output "Checking existing Application folder."

	  if (Test-Path $TargetPath)
	  {
            Write-Output "Application folder $TargetPath found. Cleaning folder contents."
            Get-ChildItem -Path $TargetPath | Remove-Item -Recurse -Force
	  }
	  else
	  {
            Throw "Application folder not found."
	  }

	  # Update application and configuration
	  Write-Output "Updating application at $TargetPath from $SourcePath."

	  Get-ChildItem $SourcePath -Filter *.zip -Recurse | Expand-Archive -DestinationPath $TargetPath
	script: Copy-Item $(zipFileOnBuildMachine) -Destination "${{parameters.artifactDirectory}}" -ToSession $session -Recurse
	displayName: Update the application

	Invoke-Command @copy
  displayName: Update the application

Restarting the pool

Similar to stopping, we call the IISAdministration module to start the pool

- template: /steps/powershell-on-target-machine.yml
  parameters:
	username: $(username)
	password: $(userPassword)
	jeaConfig: ${{parameters.jeaConfig}}
	machineName: $(machineName)
	scriptBlock: '& C:\ss\scripts\v8\start-iis-app-pool.ps1 -PoolName $(appPoolName)'
	displayName: Start application pool

And the corresponding start-iis-app-pool.ps1:

[CmdletBinding()]
param(
    [Parameter(Mandatory = $true)]
    [string]$PoolName
)

$appPool = Get-IISAppPool -Name $PoolName;
if ($appPool.State -eq "Started")
{
    Write-Output "Application pool $AppPoolName is already started."
    return
}

Write-Output "Starting Application pool $AppPoolName"
try
{
    $appPool.Start()
}
catch
{
    Write-Output $appPool
    Throw "Failed to start Application pool $AppPoolName. Error: $_"
}

# wait until the pool has actually started
$timeout = 20  # Timeout in seconds
$elapsed = 0
while ($appPool.State -ne "Started" -and $elapsed -lt $timeout)
{
    Start-Sleep -Seconds 2
    $elapsed += 2
}
if ($appPool.State -ne "Started")
{
    Write-Output $appPool.State
    Throw "Application pool $AppPoolName could not be started within the timeout period."
}

Complete

Here's the completed pipeline:

# ASP.NET Core Web application on-premises update script

# Requires these variables, defined in parameters.deploymentVariableGroups:
# username - deployment user credentials
# userPassword
# machineName - the target server
# appPoolName - the application pool to manipulate

parameters:
  #! Generic pipeline parameters
  - name: artifactName
    type: string
    default: drop

  # The name of the Azure Pipelines environment targeted by the update
  - name: deploymentEnvironmentName
    type: string

  # The name of the JEA config which is providing the installation user the proper access
  - name: jeaConfig
    type: string

  # Names of the variable group(s) containing the deployment configuration
  - name: deploymentVariableGroups
    type: object

  - name: pool
    type: string
    default: ''

  - name: deploymentName
    type: string
    default: DEPLOY_DOTNET

  # where application should be deployed to. Local path in the target machine, e.g. S:/wwwroot/myapp
  - name: appDirectory
    type: string

  # Path where the artifact is created on the target machine
  - name: artifactDirectory
    type: string

  - name: dependsOn
    type: object
    default: []

jobs:
  - deployment: ${{ parameters.deploymentName }}
    ${{ if not(eq(length(parameters.dependsOn), 0))}}:
      dependsOn: ${{ parameters.dependsOn }}
    workspace:
      clean: all
    displayName: Update application on on-prem IIS (${{ parameters.deploymentEnvironmentName }})
    environment: ${{ parameters.deploymentEnvironmentName }}
    ${{ if parameters.pool }}:
      pool: ${{parameters.pool}}
    variables:
      - ${{ each g in parameters.deploymentVariableGroups }}:
          - group: ${{g}}
      - name: zipFileOnBuildMachine
        value: $(Pipeline.Workspace)/${{ parameters.artifactName }}
      - name: artifactZipPath
        value: $(Pipeline.Workspace)/${{ parameters.artifactName }}/**/*.zip

    strategy:
      runOnce:
        deploy:
          steps:
            - checkout: none
            - download: none
            - task: DownloadPipelineArtifact@2
              displayName: Download build artifact
              inputs:
                source: current
                runVersion: latest
                artifactName: ${{ parameters.artifactName }}
                targetPath: $(zipFileOnBuildMachine)

            - template: /steps/powershell-on-target-machine.yml
              parameters:
                username: $(username)
                password: $(userPassword)
                machineName: $(machineName)
                scriptBlock: |
                  if (Test-Path -Path "${{parameters.artifactDirectory}}") {
                    Remove-Item -Path "${{parameters.artifactDirectory}}" -Recurse -Force
                  }
                script: Copy-Item $(zipFileOnBuildMachine) -Destination "${{parameters.artifactDirectory}}" -ToSession $session -Recurse
                displayName: Transfer artifact to target machine

            - template: /steps/powershell-on-target-machine.yml
              parameters:
                username: $(username)
                password: $(userPassword)
                jeaConfig: ${{parameters.jeaConfig}}
                machineName: $(machineName)
                scriptBlock: '& C:\scripts\shut-down-iis-app-pool.ps1 -PoolName $(appPoolName)'
                displayName: Stop application pool

            - template: /steps/powershell-on-target-machine.yml
              parameters:
                username: $(username)
                password: $(userPassword)
                machineName: $(machineName)
                scriptBlock: |
                  $TargetPath = "${{parameters.appDirectory}}"
                  $SourcePath = "${{parameters.artifactDirectory}}"
                  Write-Output "Checking existing Application folder."

                  if (Test-Path $TargetPath)
                  {
                      Write-Output "Application folder $TargetPath found. Cleaning folder contents."
                      Get-ChildItem -Path $TargetPath | Remove-Item -Recurse -Force
                  }
                  else
                  {
                      Throw "Application folder not found."
                  }

                  # Update application and configuration
                  Write-Output "Updating application at $TargetPath from $SourcePath."

                  Get-ChildItem $SourcePath -Filter *.zip -Recurse | Expand-Archive -DestinationPath $TargetPath
                script: Copy-Item $(zipFileOnBuildMachine) -Destination "${{parameters.artifactDirectory}}" -ToSession $session -Recurse
                displayName: Update the application

            - template: /steps/powershell-on-target-machine.yml
              parameters:
                username: $(username)
                password: $(userPassword)
                jeaConfig: ${{parameters.jeaConfig}}
                machineName: $(machineName)
                scriptBlock: '& C:\scripts\start-iis-app-pool.ps1 -PoolName $(appPoolName)'
                displayName: Start application pool

Conclusion

Since we're really not doing anything complex here, it's quite surprising how complex this all looks. Most of the lines are wasted on handling the PowerShell sessions, while the functionality in each step itself is very simple.

This all seems simple in retrospect, but it has been a grueling climb to get here. The biggest hurdles have certainly been in getting a reasonably secure PowerShell remoting going. I think it's very dangerous that the default PowerShellOnTargetMachines task requires an administrator account, which makes the whole security model feel like a joke. Thankfully there's this workaround, but it should not be necessary at all.

Hope this helps you get to a working setup faster than we did!

Note: I only came up with the powershell-on-target-machine.yml while writing this post. Since for the post I had to attempt to put everything in a single file, the amount of ps-boilerplate started to hurt badly. As always, there is a certain pain threshold you have to reach before making a change, and this was it.

Did I miss anything? Thoughts, comments? Send me an email!

#automation #azure #devops #tech