Continuous Integration

Dagger & Agola seem nice.

Notes

• Fully-automated seamless CI/CD is the goal.
• At Stripe my PR is often reviewed within 10 minutes. That makes a huge difference for shipping fast.
• I feel like if I’m sitting on someone’s PR, I’m blocking them from moving forward. Rarely the case, but it’s a good motivator.
• The real missing part for CI/CD automation is a local development experience. Sounds counter-intuitive but being able to run the E2E CI/CD pipeline locally and validate failures saves engineers from the 20 minute push, coffee, failure, fix loop.
• For sane development, it’s super-duper important that you don’t CI the PR branch itself (which might be stale), but rather CI the result of incorporating PR into master, before atomically resetting master to the resulting CIed commit.

Links

• Concourse CI - Automation system written in Go. It is most commonly used for CI/CD, and is built to scale to any kind of automation pipeline, from simple to complex. (Web)
• HN: Jenkins Is Getting Old (2019)
• Buildkite - Platform for running fast, secure, and scalable continuous integration pipelines on your own infrastructure. (CLI) (Docs) (Docs Code)
• Buildkite Agent - Open-source toolkit written in Golang for securely running build jobs on any device or network.
• Buildkite Builder - Ruby gem for programmatically creating Buildkite pipeline YML.
• Buildkite Terraform Provider
• buildx - Docker CLI plugin for extended build capabilities with BuildKit.
• Lobsters: What CI platforms do you use? (2019)
• huskyCI - Performing security tests inside your CI.
• Strider - Open Source Continuous Integration & Deployment Server. (Web)
• Sleuth - Bring full visibility and reporting across your remote team's DevOps stack.
• Snyk - Enables businesses to easily build security into their continuous development process. (Community) (Code)
• See 13 Netlify features for the best control of development workflow (2020)
• Netlify Build - Pluggable tool for running builds locally and in Netlify CI.
• Netlify Build Plugins
• GitOps Engine - Reusable library that implements core GitOps features.
• Automating safe, hands-off deployments (2020)
• Platform.sh - End-to-end cloud PaaS for hosting, development, and deployment. (Docs)
• Drone CI - Automate Software Testing and Delivery.
• PipeCD - Continuous Delivery for Declarative Kubernetes, Serverless and Infrastructure Applications. (Web)
• Conducto - Radically Better CI/CD. Python, not YAML. Lightning quick debugging. Seamless GitHub and Slack integration. (Lobsters)
• Waypoint - Build, deploy, and release applications across any platform. (Code) (Article) (HN) (Waypoint Plugin SDK) (Waypoint Example Apps) (Waypoint Horizon API) (Waypoint Plugin Kustomize)
• Buddy - DevOps Automation Platform.
• CDS - Enterprise-Grade Continuous Delivery & DevOps Automation Platform written in Go. (Web)
• A tiny CI system (2020) (Lobsters)
• CI for performance: Reliable benchmarking in noisy environments (2020) (Lobsters)
• Quad CI - CI server written in Simple Haskell.
• CI/CD Workflow for AWS ECS via Terragrunt and GitHub Actions (2020)
• CI/CD for ARM Development (2021)
• Abstruse CI - Free and open-source CI/CD platform for companies and individuals who value independence and simplicity. Written in Go. (Web)
• Fulfilling the promise of CI/CD (2021) (HN)
• tbls - CI-Friendly tool for document a database, written in Go.
• Setting up a CI system part 1: Preparing your test machines (2021)
• Delta CI - Bare Metal CI/CD with much faster builds. (HN)
• ProgressiveDeliveryConf 2021 - Learn how to go beyond CI/CD.
• The Tao of Continuous Integration (2021)
• Speeding up Docker builds in CI with BuildKit (2021)
• Continuous Integration With Interactive Notebooks
• Modern CI is Too Complex and Misdirected (2021) (Lobsters) (HN)
• Ask HN: How Long Is Your CI Process? (2021)
• Mega-Linter - Open-Source tool for CI/CD workflows that analyzes consistency and quality.
• Woodpecker - Simple CI engine with great extensibility. (Docs)
• BuildKit integration for Rust
• DevOps CI/CD Explained in 100 Seconds
• Werft - Kubernetes-native CI system. It knows no pipelines, just jobs and each job is a Kubernetes pod. (Code)
• How to Learn CircleCI in Simple Words (2021)
• Deliveroo CircleCI helper image - Container with useful helpers for CircleCI 2.0.
• Vela - Pipeline Automation (CI/CD) framework built on Linux container technology written in Go. (Docs)
• CircleCI CLI - Use CircleCI from the command line.
• Tekton Pipelines - Provides k8s-style resources for declaring CI/CD-style pipelines. (Web) (Buildkit Tekton)
• Tekton Chains - Kubernetes Custom Resource Definition (CRD) controller that allows you to manage your supply chain security in Tekton.
• Bitrise - Mobile Continuous Integration and Delivery. (GitHub) (Bitrise Step Libs)
• Bitrise (offline) CLI (Web)
• Zero Downtime Deployments (2021)
• Common Threat Matrix for CI/CD Pipeline
• Visual Studio App Center SDK for iOS and macOS
• App Center fastlane plugin
• snowpatch - Enabling continuous integration for patch-based development workflows.
• ci-env - Environment variables exposed by CI tools.
• go-circleci - CircleCI API (V2) Client Library in Go.
• bkl - Run Buildkite Pipelines locally.
• Example Pipelines | Buildkite
• What we learnt by migrating from CircleCI to Buildkite (2021)
• Spinnaker - Open-source continuous delivery platform for releasing software changes with high velocity and confidence. (Web)
• Danger JS - Runs after your CI, automating your team's conventions surrounding code review.
• Faster Gitlab CI/CD pipelines (2021) (HN)
• Developing Databricks' Runbot CI Solution (2021)
• Conform - Policy enforcement for your pipelines.
• YourBase - New build tool optimized for local + remote development. (Code)
• Hercules CI Agent - Runs on your infrastructure and executes tasks (evaluation, builds, etc) for your Hercules CI account.
• Bob the Builder - What CI/CD should've been. (Web)
• Cirrus CI - Modern Continuous Integration system built for the era of cloud computing. (GitHub) (CLI) (Docs Code)
• Awesome CI/CD Security
• List of Continuous Integration services
• Lightning-fast CI with nixbuild.net (2022)
• Homu - Bot that integrates with GitHub and your favorite continuous integration service.
• Dagger - Build powerful CI/CD pipelines quickly, then run them anywhere. (Code) (Intro) (HN) (Rust SDK)
• We don’t use a staging environment (2022) (HN)
• Spacelift - Flexible CI/CD for Infrastructure as Code. (GitHub) (CLI) (HN)
• Creating one CI to rule them all, with Fedor Korotkov, founder and CTO of Cirrus Labs (2022)
• Polar Signals - Continuous profiling project for applications and infrastructure. (GitHub)
• Zadig - Cloud native, distributed, developer-oriented continuous delivery product.
• Tekton Results - Aims to help users logically group CI/CD workload history and separate out long term result storage away from the Pipeline controller.
• Ratchet - Tool for improving the security of CI/CD workflows by automating the process of pinning and unpinning upstream versions.
• CI Free Tier Showdown (2022)
• goldboot - Provisioning bare-metal with CI.
• Scribe - Framework to write flexible CI pipelines in Go that have consistent behavior when ran locally or in a CI server.
• Ask HN: CI in 2022
• Actuated - Brings blazingly fast, secure builds to self-hosted CI runners.
• Bencher - Continuous benchmarking tools designed to help catch performance regressions in CI.
• Blazing fast CI with MicroVMs (2022) (HN)
• Run separate Buildkite pipelines for each folder in your monorepo
• Agola - CI/CD Redefined. (Docs)
• Datadog CI - Use Datadog from your CI.
• Continuous Benchmarking, Done Right - Run benchmarks as part of your continuous integration ⇒ automatically flag MRs which regress performance.
• Feature Flags in a CI Pipeline (2023) (HN)
• Open source Python project CI pipeline (2023) (Lobsters)
• An Ideal CI/CD System (2022)
• Go CI - High-level CI config DSL written in Go based on Dagger.
• Macige - CI workflow generator for mobile app development.