Setting up CI/CD on GitLab for Deploying a Python Flask App on Heroku

Continuous integration (CI) and continuous delivery/deployment (CD) have become essential for modern software teams looking to rapidly deliver high quality applications. This comprehensive, advanced-level guide will explain how to set up a full CI/CD pipeline for a Python Flask web application, using GitLab as the code repository and Heroku as the hosting platform for deployment.

CI/CD Benefits and Industry Adoption

Here are some key benefits driving adoption of CI/CD pipelines:

Faster Feedback Cycles – Developers get feedback sooner when bugs are caught early through automated testing. Issues can be fixed right away without slowing down releases.

Reduced Risk – Automation reduces chances of human error that could happen in manual deployments. Rigorous testing also lowers likelihood of bugs reaching production.

Improved Productivity – Time is saved from manual busywork, allowing teams to deliver features and fixes faster. Developers can focus on writing code rather than builds.

Consistency – Automated environments, dependencies and configs ensure consistency across development, testing and production.

According to the State of DevOps Report 2020 by Puppet:

• 96% of elite performers are using CI/CD practices
• Teams applying CI/CD deliver 200x more frequent deployments
• They experience 2,555x faster lead time from commit to deploy

Clearly there are tremendous productivity gains and velocity improvements from implementing CI/CD pipelines.

Choosing GitLab Over Jenkins for CI/CD

When evaluating continuous integration tools, Jenkins and GitLab CI are two of the most popular options.

Jenkins has traditionally been used for CI/CD, providing very extensive plugin support and customization. However, it can be challenging to scale and manage at enterprise levels.

GitLab CI is newer but has gained significant popularity among development teams because it offers these advantages:

• Natively integrated with GitLab source code management
• Simple YAML configuration over Jenkins‘ GUI interface
• No need to manage separate Jenkins servers and infrastructure
• More granular permissions aligned with repository contributors
• Built-in integration with docker registries
• Review Apps and Environments enable gradual rollouts

Based on the 2020 State of CI/CD report, 56% of teams now choose GitLab CI while only 16% use Jenkins, indicating it is becoming the tool of choice for CI/CD pipelines.

Let‘s look at some popular open source Python projects leveraging GitLab CI:

For demonstrating CI/CD workflows, Flask is an ideal example application given its simplicity and ease of integration.

Now we‘ll explore how to set up CI/CD for a Flask project with GitLab.

Step 1 – Set Up a Python Flask Application

We‘ll use Flask, a lightweight Python web framework:

from flask import Flask
app = Flask(__name__)

@app.route("/")
def home():
    return "Hello, World!"

if __name__ == "__main__":
    app.run()

Make sure Python 3 and Flask are installed in your environment.

Initialize Git version control locally:

git init

Next set up a virtual environment to isolate app dependencies:

python3 -m venv venv

Activate the virtual env then install packages:

source venv/bin/activate
pip install flask gunicorn

Freeze app dependencies to requirements.txt:

pip freeze > requirements.txt

This captures the exact versions for CI/CD reproducibility later.

Commit the code, requirements and files to the local repository:

git add .
git commit -m "Initial commit"

The Flask app is now ready! Next we‘ll set up the GitLab repository.

Secure Credentials Management for CI/CD Pipelines

When setting up CI/CD, security should always be front and center from the start. Some best practices include:

• Store confidential keys, tokens and passwords in protected environment variables instead of directly in config files or source code. GitLab CI allows securing variables at the project or group level.

• Scope access to these variables only to jobs that require them, not all jobs

• Revoke access to variables when people leave teams or projects

• Rotate variable values periodically

• Use RBAC to restrict access to pipelines so that only authorized team members can configure or run jobs and access logs

• Scan container images used in jobs and verify contents match expectations

• Enable 2FA for all accounts that can access production infrastructure and secrets

Following these guidelines will help limit opportunities for credentials to be accidentally exposed or misused.

Now let‘s dive into the CI/CD configuration…

Step 2 – Create a GitLab Repository

First, sign up for a free account on GitLab.

Then create a new project which will create a blank repository:

Copy the remote URL shown on the next page. Link your local repo to this new GitLab project:

git remote add origin <remote_url>
git push -u origin master

Refresh your project page and you should now see the committed files there.

Step 3 – Link GitLab to Heroku

Heroku provides a simple platform for deploying web apps. To get started:

1. Create a new Heroku app from their dashboard:

2. Install the Heroku CLI locally:

brew install heroku/brew/heroku  

3. Login and add remote to your app:

heroku login
heroku git:remote -a <app_name>

This links the Heroku app instance to your local Git repo.

Step 4 – Configure CI/CD Pipeline on GitLab

Now we will set up pipeline automation using GitLab‘s built-in CI/CD capabilities.

Understanding GitLab CI Configuration (.gitlab-ci.yml)

This YAML file defines the continuous integration pipeline:

stages:
  - build
  - test 
  - deploy

build: 
  stage: build
  script:
    - pip install -r requirements.txt

It consists of:

stages – High level groups of jobs to be run in sequence or parallel

jobs – Specific tasks to be executed as part of a stage

script – Contains the runtime commands for that job

Jobs will automatically trigger whenever code is pushed. Let‘s customize the pipeline for Python and Heroku.

Configuring Python CI Jobs

Jobs will run in isolated Docker containers by default. So first specify the Python image:

image: python:3.7 

Use pip and requirements.txt for handling dependencies:

build:
  stage: build
  image: python:3.7  
  script:
    - pip install -r requirements.txt

Running Tests in CI Pipelines

Quality CI/CD pipelines always incorporate automated testing. For Python, we can use the built-in unittest module:

test:
  stage: test
  image: python:3.7
  script:  
    - python -m unittest discover tests

This will run all test_* files under the tests/ directory after builds complete.

Aim for 90%+ test coverage of all application code, tracked in CI with tools like coverage.py.

Deploying to Heroku

The final stage deploys app changes live following successful builds and tests:

deploy_heroku:
  image: ruby:2.6  
  stage: deploy
  script:
    - apt-get update -qq && apt-get install -qq -y ruby-dev
    - gem install dpl
    - dpl --provider=heroku --app=$HEROKU_APP_NAME --api-key=$HEROKU_API_KEY
  only: 
    - master

Heroku deployment uses the dpl Ruby library.

Credentials should be stored as secret variables in your GitLab project.

Deployments only trigger from the master branch by default. Feature branches can be deployed to separate ephemeral environments for early testing before merging to master.

Defining CI/CD Environments on GitLab

Instead of having just a single pipeline, GitLab supports breaking up jobs into environments – different worlds for each stage of software:

For example:

stages:
  - test
  - build
  - review
  - staging
  - production

test:
  stage: test
  script: 
    - pytest tests/

build:
  stage: build 
  only:
    - master 
    - merge_requests

deploy_review:
  stage: review
  script:
    - deploy.sh  
  environment:
    name: review/$CI_COMMIT_REF_NAME
    url: https://$CI_COMMIT_REF_NAME.example.com
  only:
    - branches

deploy_staging:
  stage: staging
  script:
    - deploy.sh
  environment: 
    name: staging
    url: https://staging.example.com
  only:
    - master

deploy_prod:
  stage: production
  script:
    - deploy.sh
  environment:
    name: production  
    url: https://example.com
  when: manual 

This pipelines deploys branches to review environments for testing, master goes through staging then finally production which is a manual process. Environments isolate change rollouts across concerns.

Automating Releases with GitLab CI/CD

Sophisticated teams often adopt GitOps practices for application deployments, leveraging Git for release automation.

Instead of running git push heroku master to deploy, this flow is used:

1. Code lands on master branch
2. GitLab CI build, tests and packages application
3. Application version gets released to the staging environment
4. Manual approval promotes staged version to production
5. Git push automatically triggers deployment to prod

This enables staged rollouts and approval gates between environments, reducing risks and adding rollback capabilities.

Common strategies include blue-green and canary releasing. GitLab Environments provide native support for these release patterns.

Review Apps and Live Previews with GitLab CI

Review Apps provide live previews of changes from feature branches:

Whenever a merge request is created, a complete separate copy of the app is spun up to showcase that branch‘s code.

Once approved and merged to master, GitLab automatically closes down this environment. This facilitates early testing collaboration across teams.

Enabling Review Apps requires additional infrastructure components – consult the Review Apps Implementation Guide for more details.

Automating Database Migrations

For data-driven applications, the CI/CD pipeline needs to also handle any database schema and data migrations across environments.

Common approaches include:

• Run migration management commands like Flask-Migrate‘s db upgrade as jobs execute

• Run test suites against the migrated database for validation

• Package migration code with application code so environments remain in sync

• Utilize stateful data containers for easy portability across environments

Effective database change management ensures continuity across builds.

Conclusion

In closing, implementing continuous integration, delivery and deployment is vital for scaling the pace of innovation for modern software teams.

Automating testing, release and deployment processes through GitLab CI liberates developers from manual busywork to focus on features.

This guide walked through a reference pipeline for a Python Flask web application. We explored various customization options, stages and triggers across a spectrum of environments right up to production.

As covered, there are also many considerations around security, testing rigor and release patterns that teams should factor in as CI/CD pipelines mature.

Leveraging GitLab CI/CD best practices pays dividends in throughput, stability and velocity over time. The automation capabilities add tremendous speed without compromising quality or control.

So tackle the key first steps integrating CI/CD now, then incrementally expand scope over time as needed. Just don‘t deploy the same way manually ever again!