Automatic package updates in Azure Pipelines
If you want automatic package updates on Azure Pipelines but don't want to rely on community extensions for Dependabot, here's a simple way to get most of the way by yourself
Utilizing the standard update tools from your package manager can give you a good enough automatic package update pipeline. It is of course very bare-bones when compared to the functionality of Dependabot, but it might just be enough until official support maybe one day arrives.
In this post I'll showcase the functionality for npm and nuget, but naturally it will work for all package managers. The pipelines showcased here are available in the blog repo
The process is really straightforward:
- Update the packages using your package manager
- Create a new branch and push the changes
- Create a pull request
Creating pull requests
Since there isn't a built-in function for this, I wrote a post showing a template for making pull requests. The templates here will be referring to this as /internal/shared/steps/create-pull-request.yml.
Npm
Using npm install makes this part trivial. First the template:
# public/node/jobs/update-npm-packages.yml
parameters:
# the branch that the PR is created against. Will use the repo default branch if empty
- name: targetBranch
type: string
default: ''
- name: workingDir
type: string
default: './'
- name: nodeVersion
type: string
default: '>=20'
jobs:
- job:
variables:
branch_name: npmupdates/$(Build.BuildNumber)
steps:
- checkout: self
persistCredentials: true
- task: NodeTool@0
inputs:
versionSpec: ${{ parameters.nodeVersion }}
- task: Npm@1
displayName: 'NPM: update --save --force --dev'
inputs:
command: custom
customCommand: update --save --force --dev
workingDir: ${{ parameters.workingDir }}
- bash: |
git config --global user.email "some@email"
git config --global user.name "Rewiring"
git switch -c ${{ variables.branch_name }}
git commit -am "infra: update packages" || echo "##vso[task.setvariable variable=NO_CHANGES]true"
displayName: check for changes and commit
- bash: git push --set-upstream origin ${{ variables.branch_name }}
condition: ne(variables['NO_CHANGES'], 'true')
displayName: publish branch
- template: /internal/shared/steps/create-pull-request.yml
parameters:
condition: ne(variables['NO_CHANGES'], 'true')
sourceBranch: ${{ variables.branch_name }}
pullRequestDescription: 'This is an automatically created PR for updating outdated NPM packages.'
Note that we're using git commit -am "infra: update packages" || echo "##vso[task.setvariable variable=NO_CHANGES]true" to detect if any changes have been made and skip all the remaining tasks if not.
Using the template:
trigger: none
schedules:
- cron: 0 3 3 * *
displayName: on the third day of month
branches:
include:
- main
always: true
resources:
repositories:
- repository: templates
type: git
name: Rewiring
ref: refs/heads/release/v1
jobs:
- template: public/node/jobs/update-npm-packages.yml@templates
# parameters:
# fill as needed
Simplicity itself! Since npm is not really my zone, I've not done anything to the PR description. You could include a list of the updated packages in some way, which I'll show for dotnet below.
Nugets
The options for updating nuget packages are much narrower than in npm land. Mostly this is due to no support for version ranges, which kind of limits the usability of automatic tools.
dotnet-outdated makes a good effort for bridging the gap. It offers functionality for detecting outdated packages and updating them by rules, e.g. 'only update this package by minor versions'.
Here's the template, from which I've cleaned out stuff like caching and internal artifact repositories to keep it as simple as possible.
# public/dotnet/jobs/update-dotnet-nugets.yml
parameters:
# the branch that the PR is created against. Will use the repo default branch if empty
- name: targetBranch
type: string
default: ''
# names or partial names of packages that should not be upgraded to a new major version
- name: majorVersionLocked
type: object
default:
- Microsoft.EntityFrameworkCore
- Microsoft.AspNetCore
jobs:
- job:
displayName: Update outdated nugets
variables:
branch_name: nugetupdates/$(Build.BuildNumber)
outputMinor: $(Build.ArtifactStagingDirectory)/outputMinor.csv
outputMajor: $(Build.ArtifactStagingDirectory)/outputMajor.csv
outputCombined: $(Build.ArtifactStagingDirectory)/output.md
steps:
- checkout: self
persistCredentials: true
- task: DotNetCoreCLI@2
displayName: install dotnet-outdated-tool
inputs:
command: custom
custom: tool
arguments: update --global dotnet-outdated-tool --ignore-failed-sources
# restore nuget packages
- task: DotNetCoreCLI@2
displayName: dotnet restore
inputs:
command: restore
projects: '**/*.csproj'
# since we want to not upgrade major versions of some packages, we need to run two separate dotnet outdated runs
- task: DotNetCoreCLI@2
displayName: upgrade outdated nugets (version lock - Major)
retryCountOnTaskFailure: 2 # dotnet-outdated has a timeout of 20 seconds which we occasionally hit in the pipelines
inputs:
command: custom
custom: outdated
arguments: ${{ parameters.targetPath }} --upgrade --no-restore --version-lock Major --include ${{ join(' --include ',parameters.majorVersionLocked) }} -o $(outputMinor) -of Csv
- task: DotNetCoreCLI@2
displayName: upgrade outdated nugets (non-Microsoft)
retryCountOnTaskFailure: 2 # dotnet-outdated has a timeout of 20 seconds which we occasionally hit in the pipelines
inputs:
command: custom
custom: outdated
arguments: ${{ parameters.targetPath }} --upgrade --no-restore --exclude ${{ join(' --exclude ',parameters.majorVersionLocked) }} -o $(outputMajor) -of Csv
# Translate the output csv files into markdown
- bash: |
echo "|Project|Package|Old Version|New Version|" > $OUTPUTCOMBINED
echo "|-------|-------|-----------|-----------|" >> $OUTPUTCOMBINED
ls $(Build.ArtifactStagingDirectory)/output*.csv | xargs -n 1 tail -n+2 | sort -t, -k1,1 | awk -F, '{print "|"$1"|"$4"|"$5"|"$6"|"}' >> $OUTPUTCOMBINED
displayName: Combine output and convert to markdown
- bash: |
git config user.email "some@email.com"
git config user.name "Rewiring"
git switch -c ${{ variables.branch_name }}
output=$(<$OUTPUTCOMBINED)
# pull request description cannot be over 4000 characters
combinedOutput=$(echo "$output" | head -c 3950)
git commit -a -m "infra: update nugets" -m "$combinedOutput" || echo "##vso[task.setvariable variable=NO_CHANGES]true"
displayName: commit
- bash: git push --set-upstream origin ${{ variables.branch_name }}
displayName: publish branch
condition: ne(variables['NO_CHANGES'], 'true')
- template: /internal/shared/steps/create-pull-request.yml@templates
parameters:
condition: ne(variables['NO_CHANGES'], 'true')
sourceBranch: ${{ variables.branch_name }}
pullRequestTitle: 'infra: update nuget packages'
A lot more going on there, but the main process is still simple. Since we don't have native package ranges, I've showcased how you can separate minor and major updates using two separate calls to dotnet outdated. If you're using package.lock.json -files you have to restore the packages after the update has completed, or the lock files are not updated.
An interesting tangent - you can chain -m parameters in git commit -m "first" -m "second". The first one is the title, the later ones become the description.
You can use the markdown output to get a nice PR message, showing which packages were updated
Conclusions
By leveraging just your package manager it is possible to get a decent automatic package update flow going.
If you need more, you could go with the DevOps extension or try using e.g. this script for running Dependabot via containers. If, however, you're happy with the philosophy of 'simple tools for simple jobs', then the ones presented here should be decent (and fast!) enough.
Thoughts, comments? Send me an email!