Github Actions on the test rig
Aurélien Le MassonIvan Poiraudeau5 min read
As of November 13, 2019, GitHub Actions became publicly available. We have tested the automation solution from GitHub, often sold as a new CI/CD pipeline alternative. Let’s see how it fares compared to its competitors, and whether you should switch to it.
1. What GitHub Actions brings to the table
A new approach to building your CI/CD pipeline
This new product is quite different from the features of common CI/CD providers (GitLab CI/CD, CircleCI, Travis CI…). Instead of offering only virtual machines for you to run your pipeline on, GitHub chose to offer both the virtual machines and pre-built blocks of CI/CD pipelines, called Actions. The goal of an Action is to save you from coding parts of your pipeline. They can be picked up in the marketplace or you can build your own, and reuse them on several of your repositories.
These can be common actions you would find in any Pipeline (Linting, Jest testing, Uploading items to an S3 bucket), and are almost plug and play. Other Actions could replace the sometimes frustrating WebHooks:
- Updating your Codecov.io coverage
- Automatically merging / asking for review on / tagging your PR
- Sending Slack notifications
- And more
Those actions are built by the community and are open source. You can fork them at will or create your own.
This system of sharing parts of a CI/CD pipeline also exists in other CI Providers (e.g. CircleCI Orbs), but GitHub took the “sharing” part to the next level by allowing you to share actions via their Marketplace, via Docker images, or via other GitHub repositories.
A competitive pricing
Another strong selling point for GitHub Actions is its pricing. They offer 20 free parallel builds and unlimited build minutes for every open source project.
For private repositories, with the free plan you will get 2’000 free minutes (~33 hours) of build time per month, on par with GitLab free tier. Compared to the 250 free minutes of CircleCI, or 100 free builds of TravisCI, GitHub Actions and GitLab are ahead of their competitors on the free tier offer. Just like GitLab CI, you can also set up GitHub Actions on premise for free.
If you have subscribed to a paid plan, the number of minutes is even higher (up to 50’000 minutes per month for the Enterprise plan).
And a great integration with other GitHub services
One thing we loved while playing with GitHub Actions is the smooth integration it offers with other GitHub features. You can trigger Actions and Workflows with pretty much any GitHub event: a classic opening of a PR, a push of a new commit, but also an issue opening, a comment, a PR review… you get the point.
You also get an easy access to the full context of the PR (or issue, etc.) which triggered the action, and to the JavaScript GitHub API thanks to the github-script action, which lets you write data back to GitHub. These two combined allow for a wide range of automation possibilities.
2. GitHub Actions drawbacks and teething issues
GitHub Actions still has a few limitations compared to its competitors, which makes it less adapted to heavy CI/CD pipelines. But the GitHub team is closely listening to its users and is rapidly bridging the gap: for example, the much-wanted dependency caching feature was developed just before the end of GitHub Actions beta phase.
No availability for legacy pricing plans
If your organization has decided to stick to legacy pricing plans because you get more bang for the buck from them, then you’re out of luck: GitHub Actions will simply not run on any of your repos, even the public ones.
Performance of similar operations falls behind other providers
We tested GitHub Actions against CircleCI, and we found the same operations were 30% to 50% slower on GitHub Actions compared to CircleCI (though we couldn’t fathom why). Here are tests we ran:
Setup
- CircleCI: default Docker executor of free plan (medium size: 2 vCPUs, 4GB RAM)
- Github Actions: Default machine (at the time of testing: 2-core CPU, 7GB RAM)
- Image for both: Ubuntu 18.04
- No caching
Results
| Project | Command | Avg Github Actions duration | Avg CircleCI duration |
|---|---|---|---|
| yarn | yarn && yarn build && yarn lint | 1 min 32 s | 0 min 52 s |
| prettier | yarn install | 0 min 37 s | 0 min 18 s |
| prettier | yarn lint | 0 min 31 s | 0 min 15 s |
| prettier | yarn test | 4 min 02 s | 3 min 07 s |
No schematic view of workflows
Mature CI providers such as CircleCI usually provide a schematic view of workflows that helps you know if you’ve assembled your jobs well. GitHub Actions doesn’t provide such a view for the moment.
Additionally, the UI is lacking a few features such as branch or event filtering.
A view of a CircleCI workflow. Hopefully you won’t get these jobs tangled like your earphone cords.
Note: workflow view was previously available when GitHub Actions were in beta, but was removed when GitHub switched to YAML-based workflow.
A few glitches here and there
During our tests, we noticed a few workflows that would be stuck at startup or would not be displayed in the interface, or their loader would keep spinning even when the jobs were finished. Possibly, these are simply teething problems that the GitHub team will soon take care of.
File caching is limited to 400 MB per individual cache (after compression) - UPDATED
Updated: on January 6th, 2020, the GitHub Actions team raised the per-cache limit to 2 GB, solving most people’s needs. Original article text below:
In a CI pipeline, caches are usually used to quickly fetch builds or dependencies that you’ve installed in a previous CI run, to speed up future CI runs. However, many users are finding the 400mb limit too small for their uses, notably for those big node_modules folders.
At the time of writing though, the GitHub team is considering increasing this limit.
3. So, what should GitHub Actions be used for?
👍 Yay for light CIs, or for parts of your development workflows that are strongly integrated with GitHub.
🙅 Nay for heavy workflows, you may want to stick to (or start with) a more mature CI provider for now. GitHub Actions is however rapidly improving, you’d better keep an eye on it.
