I'm in the apparently small group of people who ignores 90% of the made & shared features, and push back against making internal ops to depend too much to any particular CI system.

We know how to build, package, and release our software, and at work I firmly support that we write those operations as standalone scripts that in principle could even run in the developer's machine itself. Moving them to CI is just changing the machine that will run the scripts, and that's where I draw the line of how much coupling to accept with any given runner: run scripts, inject secrets, and maybe accumulate artifacts between jobs.

There are a myriad of things done with Actions, such as preparing NPM, or running Docker containers. I see no point in them, you should be able to manually run the same commands, one by one, locally on a VM or empty Docker container, so at that point why not write them as a script. Use CI as a dumb shell runner. Getting to depend on all those small Actions saves 5 minutes today, only to make migrations immensely painful tomorrow.

This is fine if you treat your CI provider as a "dumb shell runner". But good CI platforms have actually useful features and APIs (e.g. caching) and if you want to use them, a simple Makefile isn't going to work. For projects where the difference between a cold and warm cache build is tens of minutes, those features have meaningful quality of life improvements.

This may be a tradeoff you're ok with, but for a lot of people, it's not.

C++ with templating can take a while to compile, so for caching my build system would include mounting a Docker volume with contents of ~/.ccache/. This was well integrated with the rest of the scripts, so it would work equally fine if I ran them in my laptop or in the CI runner.

My point is that I really don't believe that CI systems provide anything so unique that it couldn't be also provided by local software in a developer's laptop. If the question of "how do I build this in my laptop" is "you cannot, must use CI because we truly require some of its features", I'd consider that an ops failure.

However, I fully admit my experience comes from small to middle size projects. I cannot talk about big or huge scale projects. Maybe that's where needs grow so complex that CI features become truly necessary.

> My point is that I really don't believe that CI systems provide anything so unique that it couldn't be also provided by local software in a developer's laptop. If the question of "how do I build this in my laptop" is "you cannot, must use CI because we truly require some of its features", I'd consider that an ops failure.

So I do agree with this. But I think there's nuance here.

The job of producing a build artifact involves steps that broadly break down into two categories: setting up the context, and doing the build. I totally agree that the "doing the build" bit should be a series of simple steps that are agnostic of their environment. A shell script, or a Makefile - something you can just invoke anywhere.

But the context bit is also super-important. On my laptop, I've already got the (e.g.) right JDK installed so that when I run `make`, the build succeeds. But I'm also not wiping my laptop before every build. On a CI platform, you're effectively starting from scratch every time, so whilst you can go and write your own code to set up the caching, download the tooling etc. etc. there's an enormous amount of value to be gained by re-using the CI-platform's software and features to do that as easily as possible. No-one wants to be writing code that works out how to go set up the right JDK in the right place, when `actions/setup-java` will basically do all that for you.

In theory, yes, you could go and curate a container image that has everything you need and just run your build inside that, but now you've got two bits of software to manage.

If you can't run the build locally, then yes, you're in a pickle.

Thanks for expanding on your point of view. Now that I read you, I'd say that we fundamentally agree.

My experience has been to use carefully constructed Docker images that have the required dependencies to build the project. This removed the need for installing everything in the correct version on each dev system, and ensured a commonly shared base system on which to build stuff.

However I agree that a "context" can have lots of moving parts, and I was disregarding most of them as "exercise for the reader" aka. devs ought to know what they're doing if they wanted to build locally. CI can help a lot with that.

This is something I’ve struggled to communicate internally.

The closest I came was a GitHub actions job that did preflight checks to make sure all the right dependencies and sdks were present before calling into the same build script we use locally.

Don’t care where or how you set up your build environment, if the pfc passes your build should too. We still have some holdouts that do environment setup with a make file :/

Caching is the biggest reason why you shouldn't rely on CI platform "features" because they're quite bad at it. Roll your own and be happier with lower prices and no lock in.

I agree with this, I recently had a pretty long network issue with our CI system and for the most part we just reran scripts or converted ci yml into scripts.

I like CI for secrets management, scrpted pipelines and deploying. And having one button do do all of that. But all the extra junk people do with it seems like another thing to have break.

Indeed - a whole generation seem to be doomed to learn the hard way the painful lessons of lock-in.

Same. I've also had to do things like migrate CI when a company was acquired, from Travis to Circle and Circle to GitLab in the past. It's very painful to do this if you've leaned into everything the CI service offers.

Agree, I like to have a top level "verify" script that does all checks and have CI run exclusively and minimally that one script. Ideally the verify script is just a wrapper around a build and test command

I definitely agree with and do this — it only falls apart when CI environment stuff interferes with file locations and other external state.

That said it is nice (if the scripts are simple) to do things like machine set up very easily.

Especially if you have to run and deploy on your own servers. Not everything can be in the cloud.

> Gitlab CI is still the best CI in the game IMO,

Oh god, if this is the best CI in the game, I don't want to be part of this game anymore. I work with (as in write pipeline code for) Gitlab CI almost every day and it's absolutely horrible. Not only does YAML lack any type safety whatsoever, but every day I run into yet another open ticket in the Gitlab issue tracker because using a declarative (YAML-based) approach (as opposed to an imperative one) essentially means that the Gitlab devs need to think of every possible use case beforehand, so as to be able to define a declaration & write an implementation for it. Clearly this is impossible, so now I'm banging my head against a wall on the daily.

That’s not my experience at all. You do know you can run arbitrary shell scripts, and that means they don’t need to think of every possible use case?

If your CI process is so complex that YAML + arbitrary code doesn’t work, you might want to get that checked, it’s not normal

Random example:

I want to define a setup job to build & push a base Docker image to our container registry. I then want to use this image as IMAGE in all subsequent jobs. This is impossible because the IMAGE field in YAML cannot be dynamic (determined at runtime) but I'd like to version/tag my image using the $CI_COMMIT_SHA.

This is definitely possible. We've been building an image in one phase and then running it in subsequent phases for years with gitlab.

I think you are going about this wrong. Are you generating an image tag dynamically? When you are tagging the image, make sure that you generate the tag deterministically based on information that is available to gitlab when the pipeline is created.

So for example, you could use the tag foo:$PIPELINE_ID instead of foo:$random

My apologies, I misspoke: I wanted the image tag not to be $CI_COMMIT_SHA but to be a hash dynamically generated from certain files in the repo. The issue is that IMAGE won't accept a dynamically generated environment variable (passed from job to job via a dotenv artifact).

I think you could just jam anything you need in pipeline yaml from the dotenv file into outputs/variables then do something like this https://stackoverflow.com/a/71575683/2751619

No, this doesn't work. Also note that the StackOverflow link is about Github Actions, not Gitlab CI.

Did you know that with Gitlab you can generate gitlab ci yaml in a job runtime and then run that yaml as a child pipeline using trigger:include:artifact?

This was the only way I could create dynamic terraform pipelines which changed depending on a plan output.

I'm sure could use it to achieve what you've described.

Thank you, that's indeed a good point. And yes, I did consider that. However, then the Gitlab UI (pipelines overview etc.) ceases to be very useful as everything will be inside one big child pipeline (i.e. individual jobs will no longer be shown in the overview). My coworkers would have hated me.

Can't you use build time variables for this? https://docs.gitlab.com/ee/ci/variables/#pass-an-environment...

I tried this but it didn't work.

Addendum: To see why it doesn't work, please see https://news.ycombinator.com/item?id=38527692 (which includes a slight correction of my original comment).

The image may be determined at runtime, but it’s not required to exist until a runner picks up the job. So use the $CI_COMMIT_SHA in the image name and push the image in a job that runs before the other jobs that use the image.

You might also want to look into Downstream Pipelines.

(Please see https://news.ycombinator.com/item?id=38527692 for a correction of my original comment.)

The issue is that the IMAGE field in YAML doesn't pick up environment variables that are being passed on from a previous job via a dotenv artifact.

As for downstream pipelines, please see https://news.ycombinator.com/item?id=38525954

I am surprised to hear this as well. We had Jenkins in my previous gig. It worked, but I spend all my time keeping it humming and learnt nothing else. In my current gig, we were a Drone shop. We switched to Harness CI enterprise and it’s worked really well for us. Their hosted builds are pretty speedy!

We did evaluate Gitlab CI but went with Drone. Gitlab CI is not a top 5 CI vendor IMO.

Thanks to recognizing Harness! Full disclosure, I’m a harness employee, using Harness for CI/CD on a daily basis. Some of the best things in Harness to make us more productive at Harness, using Harness:

* Harness CI is the fastest CI solution on the market - through feature like ML-powered Test Intelligence, that allows running only the tests that are related to a code change, as well as other innovative capabilities. We use it heavily with our java applications and see test cycle reduction up to 80%. It can be used with Java, Ruby, .Net and other languages as well and the savings are significant. It also lowered our infra spent - lower build time means less build infrastructure costs

* Advanced CD: Advanced use cases like Blue/Green and Canary deployment, and rollbacks are available out of the box with Harness. No scripting is needed for implementing complex deployment use cases

* Visual pipeline editor, fully integrated with git - you can author pipeline as code in your git repo, but also - have a great authoring experience in Harness UI using yaml or visual editors. The visual editor make it super easy to understand existing pipelines as well as modifying them

* Plugins - harness support thousands of community plugins including Drone Plugins, Github Actions and Bitrise Steps in your CI Pipelines.

* Unbeatable Governance and Compliance - Harness provides robust, enterprise-grade governance for CI/CD processes. Using OPA-based policies and granular templates, customers can centrally enforce quality and security standards across all pipelines (for example, require security scans to be executed before deployment is allowed or which community plugins are allowed) .

* Reports and insights - looker based dashboard gives you many valuable reports out-of-the-box, but also the flexibly to create your own reports, so you can slice and dice the data based on your needs.

This is really just the tip of the iceberg , I encourage you to check out our website harness.io to get the full scope of our capabilities.

Cheers

I'm not following. You're surprised to hear complaints about Gitlab, even though you're not actually using Gitlab (in fact, you say "Gitlab CI is not a top 5 CI vendor IMO") and you are praising a completely different product (Harness CI)?

Oh, I meant I was surprised Gitlab is a good CI vendor. Lol.

Granted, I haven't used Gitlab CI in 5 years, but I would not praise it either. In fact, the entire Gitlab UI and UX annoyed me so much I moved our company repos to GitHub.

The latest redesign where they grouped all the sidebar links into even more categories just made it ludicrously worse.

> GitLab absolutely innovated many hard parts of CI/CD as a platform-native piece

As someone who only used GitLabs CI briefly when it initially launched, what hard parts did they innovate on exactly?

As far as I could tell, it's a run-of-the-mill CI/CD platform, for better or worse, but nothing I'd call "innovative".

But again, maybe since the first time I tried it when it launched, it has changed, and I missed something really cool they did.

Innovative is a strong word, but they always had the best support for a variety of deployment archetypes, including baremetal, kubernetes, docker, docker+machine (autoscales VMs in a public cloud to run many Docker executor runners.) They also had JWT authentication support built into GitLab before GitHub even had a CI offering at all. The GitLab kubernetes agent makes their CI/build tool more like a CD tool for organizations that primarily deploy to kubernetes clusters, which is fairly unique for a CI tool to act as a CD tool, though with scripts people often just make the CI tools do deployments anyway.

My experience is limited to GitHub Actions, GitLab CI, and Jenkins. I've never been in an organization that managed to crack the code on running Jenkins in a sane way (always a poorly maintained, bloated mess of plugins, unfortunately) so I can't say I count Jenkins as a contender in this conversation. One could argue it's unfair to discount Jenkins because of organizations that manage it poorly, but there's something to be said about a tool that is so easy to accidentally run in an unmaintainable way. I also have some limited experience with Drone and Circle, but not enough to talk about them confidently.

They integrated in-repo configuration, self-hosted runners, composable workflow definitions, a robust API, and container building and registries for the first time in a CI tool.

Of course GitHub came and blew that out of the water with a composable social actions ecosystem. Which was brilliant and remains the killer feature that puts GitHub far beyond reach of other CI platforms, especially now that they've improved in-repo composability a few months ago.

> They integrated in-repo configuration, self-hosted runners, composable workflow definitions, a robust API, and container building and registries for the first time in a CI tool.

Out of those, I think only "composable workflow definitions" would be one of the features that other CI platforms didn't have, before GitLab even existed as a project. You might want to re-read the history and features of build platforms before GitLab.

I'm very familiar with the history of CI platforms before GitLab, having used Hudson, Jenkins, Circle, Travis, Appveyor, Codebuild, and a whole bunch of other platforms extensively. While some of them had these features, none of them had all of the features, few of them meaningfully integrated the features together, and most of them provided the features in a very half-assed and unreliable way.

Off the top of my head:

- Bring your own runner is easiest/most robust

- environment deploys and management

- tons of deployment integrations

- DevSecOps features

- Built in support ticket handling

- Free container & package registry

For most of these I’m pretty sure gitlab had them built into the platform (and free!) first

It’s possible to do ‘uses’ on GitLab, but there’s a few extras needed admittedly (branch name and path to file)

    include: https://gitlab.com/awesome-project/raw/main/.before-script-template.yml

https://docs.gitlab.com/ee/ci/yaml/includes.html#include-a-s...

But I agree, the social virality is what makes GitHub Actions what it is.

I used to only visit a GL project when I needed to look through Nvidia cuda container builds… everything else I needed was on GH :shrugs:

I find GitHub Actions incredibly counter-intuitive, personally.

It’s the confusion between actions(basically third party plugins) and workflows(the ci that you are building for yourself) that got me.

I think it was a mistake for Github to name the whole product "Actions" and then re-use the same word for a specific component within system. It's really natural to say something like "we need a push action for this repo" when it might be more correct to say "a push workflow"

> I think it was a mistake for Github to name the whole product "Actions" and then re-use the same word for a specific component within system.

You're right:

- product is called GitHub Actions

- your workflow (consisting of jobs and steps) is a workflow

- 3rd party workflow included in yours is an action, not workflow though

- I more often hear engineers refer to workflows as actions

But there are more annoying decisions

- (org-wide) required workflows being recently deprecated

- the feature was buggy (ie used alongside org-wide branch rules)

- but could have been fixed, not deprecated

- some kind of "marketplace" (with reviews, developers trust levels etc) for modular/pluggable actions (workflows) would be welcome; currently it's a "1st solution fitting the problem used" mess with little to no standarisation

- I find the necessity to write a step cloning the repo from which a workflow is running ridiculous - it should be at most a single configuration line somewhere at the top of a workflow

> 3rd party workflow included in yours is an action, not workflow though

Not entirely accurate. You can use reusable workflows from third parties and you can use actions from third parties. It being a third party's doesn't mean it's an action. Also, you can make your own actions, it doesn't need to be a third party's.

Just to add to the naming confusion.

Thank you for finally explaining this to me (have had to deal with github ci off and on for months now).

Travis and Circle CI predated Gitlab, with YAML-configured workflows checked in to the repository and the ability to use custom images.

GitlabCI's innovation was integration: making it a feature of repo hosting.

I moved from Jenkins to Travis several years ago, and Travis's UI and configuration was such a fresh breath. Back then, pretty much _every_ OS project used it because of their generous offerings for open source projects. I agree with this comment that, and from my own experience living through the transitions, Travis CI made a massive leap leveling up the modern CI we have today.

Yeah I didn’t say they innovated CI/CD — I said they innovated CI/CD as a platform-native offering!

Currently working on a gitlab api thing for work, and I gotta say their API us a complete joke. GitHub runs laps around them in this area. Gitlab is actually worse than bitbucket, believe it or not.

In my own project we actively cut down on external CI deps (non-official github actions) as much as possible, it seems incredible unsafe

When we do use an external dep (like in two places for our massive project) we pin it to a specific GIT sha that we manually curated to not contain anything weird

Very good summary. I think both Gitlab CI and Github Actions are very good, but I think gitlab ci is more practical because of how easy it is to build custom advanced pipelines vs the great re-use of github actions.

How is Gitlab CI materially different from the jenkins model?

I find that the only difference is that it's YAML - so even harder to debug, and maintains the same model where you must re run an entire pipeline every commit to test functionality.

Yeah YAML isn’t ideal, but I personally found jenkins to be terrible to configure, not super well documented, missing features compared to an installation of gitlab (ex built in registry).

Also the ability to rerun, target, and influence builds themselves is better on GitLab as well I think.

You can run locally first to test functionality : - )

oh that's nice I didn't know that! Still a stickler on yaml for code though

how dare you suggest something so pedestrian.

I love GitHub actions and think gitlab is okay.

I found actions easier to use