{"id":884,"date":"2024-08-21T00:26:00","date_gmt":"2024-08-21T00:26:00","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=884"},"modified":"2024-08-21T00:26:00","modified_gmt":"2024-08-21T00:26:00","slug":"jenkins-multibranch-pipeline-tutorial","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=884","title":{"rendered":"Jenkins Multibranch Pipeline Tutorial For Beginners"},"content":{"rendered":"

If you are looking for a well-automated Pull Request based or branch-based<\/strong> Jenkins Continuous Integration<\/strong> & Delivery<\/strong> (CI\/CD) pipeline, this guide will help you get the overall picture of how to achieve it using the Jenkins multibranch pipeline.<\/p>\n

Jenkins’s multi-branch pipeline is one of the best ways to design CI\/CD workflows as it is entirely a git-based (source control) pipeline as code. This guide will talk about all the key concepts involved in a Jenkins multi-branch pipeline setup.<\/p>\n

Jenkins Multibranch Pipeline Fundamentals<\/h2>\n

Let\u2019s start with the multi-branch pipeline basics. Specifically, in this section, I will cover the concept of a multi-branch pipeline and why it is essential to use it for all Jenkins CI\/CD pipelines.<\/p>\n

I\u2019ll also show you how a multi-branch pipeline works with a detailed workflow diagram.<\/p>\n

What is a Multi-branch Pipeline?<\/h2>\n

A multi-branch pipeline is a concept of automatically creating Jenkins pipelines based on Git branches.<\/p>\n

It can automatically discover new branches in the source control (Github) and automatically create a pipeline for that branch<\/strong>. When the pipeline build starts, Jenkins uses the Jenkinsfile<\/code><\/strong> in that branch for build.<\/p>\n

SCM (Source Control) can be Github, Bitbucket, or a Gitlab repo.<\/p>\n

\"What
Click to view in HD<\/span><\/figcaption><\/figure>\n

You can choose to exclude selected branches<\/strong> if you don’t want them to be in the automated pipeline with Java regular expressions.<\/p>\n

Multi-branch pipeline supports PR based branch discovery<\/strong>. Meaning, branches get discovered automatically in the pipeline if someone raises a PR (pull request) from a branch or a forked repository. So if you are looking for a Pull Request based Jenkins build workflow, this is a great option.<\/p>\n

Also, you can add conditional logic to the Jenkinsfile<\/code><\/strong> to build jobs based on the branch requirement.<\/p>\n

For example, if you want the develop branch to run only unit testing, sonar analysis and artifact build, you can have a condition to skip the deployment stage with a when<\/code><\/strong> a condition, as shown below.<\/p>\n

\"multibranch<\/figure>\n

So whenever the code is merged develop<\/code><\/strong> branch, the pipeline will run the unit testing and sonar analysis stages skipping the deployment stage.<\/p>\n

Also, multi-branch pipelines are not limited to the continuous delivery of applications. You can use it to manage your infrastructure code as well.<\/p>\n

One such example is having a continuous delivery pipeline for Docker image builds <\/a>, terraform deployment pipeline, VM image patching, building, and upgrade process.<\/p>\n

How Does a Multi-Branch Pipeline work?<\/h2>\n

I will walk you through a basic git-flow based build and deployment<\/strong> workflow to understand multi-branch pipeline in a better way. The branching strategy provided in this example is for explanation purposes only.<\/p>\n

Let’s say I want a Jenkins pipeline to build and deploy an application with the following conditions.<\/p>\n

    \n
  1. The application repo would have two branches (main and develop)<\/li>\n
  2. Development starts with the developer forking the application repository and creating a branch in the forked repository. The developer commits code to the feature branch. Once the code is locally tested and ready for integration, they will raise a PR to the develop branch<\/strong> of the main repository from the forked repo.<\/li>\n
  3. Whenever a developer raises a PR from the feature branch of the forked repository to the develop branch, a Jenkins pipeline should trigger to merge the source and target branches<\/strong>, followed by running unit tests and static code analysis. Lets call it as PR build.<\/li>\n
  4. After the code passes the tests in the PR build, the developer or reviewer merges the PR<\/strong> into the develop branch.<\/li>\n
  5. Once the code is merged it to develop branch, it should trigger a pipeline that will run the relevant tests and deploy the code in relevant envirnments (Eg, dev, qa, etc)<\/li>\n
  6. When the code is ready for release, developers raise a PR from the develop branch to the master branch. This should trigger a PR build pipeline<\/strong> that will merge the source and target branches, run unit tests, perform code analysis, build artifacts (docker image, jar files etc), conduct vulnerability tests, etc.<\/li>\n
  7. If the tests pass, the PR will be reviewed and merged into the main branch.<\/li>\n
  8. When the merge happens, Jenkins should trigger a build that will compile the code, create a release artifact, and deploy it to the staging and pre-production environments.<\/li>\n<\/ol>\n

    From the above conditions, you can see that there is no manual trigger of Jenkins jobs<\/strong>, and whenever there is a pull request for a branch, the pipeline needs to be triggered automatically and run the required steps for that branch.<\/p>\n

    This workflow builds a great feedback loop for engineers<\/strong> and avoids dependence on the DevOps team to build and deploy in non-prod environments.<\/p>\n

    Developer can check the build status on Github and take decisions on what to do next.<\/p>\n

    This workflow can be achieved easily through a Jenkins multi-branch pipeline.<\/p>\n

    The following image shows how a multi-branch pipeline workflow would look like for the above example build process.<\/p>\n

    \"multibranch
    Click to view in HD<\/span><\/figcaption><\/figure>\n

    Here is how the multi-branch pipeline works.<\/p>\n

      \n
    1. When a developer creates a PR from a feature branch from forked repo to develop a branch, Github sends a webhook with the PR information to Jenkins.<\/li>\n
    2. Jenkins receives the PR and finds the relevant multibranch pipeline, and creates a PR build pipeline automatically. It then runs the jobs with the steps mentioned in the Jenkinsfile<\/code><\/strong>. During checkout, the source and target branches<\/strong> in the PR gets merged. The PR merge will be blocked on Github until a build status from Jenkins is returned (It Github ruleset is configured).<\/li>\n
    3. Once the build finishes, Jenkins will update the status to Github PR. Now you will be able to merge the code. If you want to check the Jenkins build logs, you can find the Jenkins build status and log link in the PR status (If Github App forJenkins is configured).<\/li>\n<\/ol>\n

      Multibranch Pipleline Jenkinsfile<\/h2>\n

      Before jumping into implementation, let’s look at multibranch pipeline Jenkins example Jenkinsfile that can be used in the pipeline.<\/p>\n

      For the multibranch pipeline to work, you need to have the Jenkinsfile in the SCM repo.<\/p>\n

      If you are learning\/testing, you can use the multibranch pipeline Jenkinsfile given below. It has a checkout stage and other dummy stages, which echoes the message.<\/p>\n

      Also, you can clone and use this Github repo<\/a> which has this Jenkinsfile.<\/p>\n

      \n
      Note<\/strong><\/b>: Replace the agent label agent-01<\/strong><\/code><\/b> with your Jenkins agent name.<\/div>\n<\/div>\n
      pipeline {\n\n    agent {\n        node {\n            label 'agent-01'\n        }\n    }\n\n    options {\n        buildDiscarder logRotator( \n            daysToKeepStr: '16', \n            numToKeepStr: '10'\n        )\n    }\n\n    stages {\n        \n        stage('Cleanup Workspace') {\n            steps {\n                cleanWs()\n                sh \"\"\"\n                echo \"Cleaned Up Workspace For Project\"\n                \"\"\"\n            }\n        }\n\n        stage('Code Checkout') {\n            steps {\n                checkout([\n                    $class: 'GitSCM', \n                    branches: [[name: '*\/main']], \n                    userRemoteConfigs: [[url: 'https:\/\/github.com\/spring-projects\/spring-petclinic.git']]\n                ])\n            }\n        }\n\n        stage('Unit Testing') {\n            steps {\n                sh \"\"\"\n                echo \"Running Unit Tests\"\n                \"\"\"\n            }\n        }\n\n        stage('Code Analysis') {\n            steps {\n                sh \"\"\"\n                echo \"Running Code Analysis\"\n                \"\"\"\n            }\n        }\n\n        stage('Deploy To Dev & QA') {\n            when {\n                branch 'develop'\n            }\n            steps {\n                sh \"\"\"\n                echo \"Building Artifact for Dev Environment\"\n                \"\"\"\n                sh \"\"\"\n                echo \"Deploying to Dev Environment\"\n                \"\"\"\n                sh \"\"\"\n                echo \"Deploying to QA Environment\"\n                \"\"\"\n            }\n        }\n\n        stage('Deploy To Staging and Pre-Prod Code') {\n            when {\n                branch 'main'\n            }\n            steps {\n                sh \"\"\"\n                echo \"Building Artifact for Staging and Pre-Prod Environments\"\n                \"\"\"\n                sh \"\"\"\n                echo \"Deploying to Staging Environment\"\n                \"\"\"\n                sh \"\"\"\n                echo \"Deploying to Pre-Prod Environment\"\n                \"\"\"\n            }\n        }\n\n    }   \n}\n<\/code><\/pre>\n
      Setup Jenkins Multi-branch  Pipeline<\/h2>\n
      Now, I will walk you through the step by step process of setting up a multi-branch pipeline on Jenkins.<\/p>\n
      This setup will be based on Github and latest Jenkins version.<\/p>\n
      If you are learning, you can use the following repository as a reference for the pipeline.<\/p>\n
      https:\/\/github.com\/techiescamp\/jenkins-pipelines<\/code><\/pre>\n
      \n
      Important Note:<\/strong><\/b> You need to create your own repository to test the multibranch pipeline.<\/div>\n<\/div>\n
      Create Multibranch Pipeline 
      (Step by Step Guide)<\/h2>\n
      Step 1: <\/strong>From the Jenkins home page create a \u201cnew item\u201d.<\/p>\n
      \"Jenkins<\/figure>\n
      Step 2: <\/strong>Select the \u201cMultibranch pipeline\u201d from the option and click ok.<\/p>\n
      \"\"<\/figure>\n
      Step 3: <\/strong>Click \u201cAdd a Source\u201d and select Github.<\/p>\n
      \"\"<\/figure>\n
      Step 4: <\/strong>Under the credentials field, select Jenkins, and create a credential with your Github username and password.<\/p>\n
      \"\"<\/figure>\n
      Step 5: <\/strong>Select the created credentials and provide your Github repo to validate the credentials as shown below.<\/p>\n
      \"validate<\/figure>\n
      Step 6:<\/strong> Under “Behaviours<\/strong>” select the required option matches your requirement. You can either choose to discover all the branches in the repo or only branches with a Pull Request.<\/p>\n
      The pipeline can discover branches with a PR from a forked repo as well. Choosing these options depends on your required workflow.<\/p>\n
      The options shown below is for the PR based workflow I explained earlier in the guide.<\/p>\n
      \"\"<\/figure>\n
      There are additional behavior you can choose from the \u201cadd\u201d button. <\/p>\n
      For example, If you choose not to discover all the branches from the repo, you can opt for the regular expression or wildcard method to discover branches from the repo as shown below.<\/p>\n
      \"Multibranch<\/figure>\n
      Here is a regex and wildcard example.<\/p>\n
      \"Multibranch<\/figure>\n
      Step 7: <\/strong>If you choose to have a different name for Jenkinsfile<\/strong>, you can specify it in the build configuration.<\/p>\n
      In the \u201cScript Path\u201d option, you can provide the required name. Ensure the Jenkinsfile is present in the repo with the same name you provide in the pipeline configuration.<\/p>\n
      I am using multi-branch-pipeline\/Jenkinsfile<\/code><\/strong> becuase the Jenkinsfile is present in the multi-branch-pipeline folder.<\/p>\n
      Also, Enable “Discard old builds” to keep only required build logs as shown below.<\/p>\n
      \"\"<\/figure>\n
      Step 8: <\/strong>Save all the job configurations.<\/p>\n
      Jenkins scans the configured Github repo<\/strong> for all the branches and PR requests based on our configurations.<\/p>\n
      The following image shows the job scanning the three branches, and since I haven\u2019t raised any pull request, Jenkins won\u2019t create any branch-based pipeline. I will show how to test the automatic pipeline creation after the webhook setup.<\/p>\n
      \"multibranch<\/figure>\n
      Till now, we have done configurations on the Jenkins side to scan branches based on the PR requests.<\/p>\n
      To have a complete workflow, we need to configure a webhook in GitHub to send all repository events (commits, PRs, etc.) to Jenkins, so the pipelines can be triggered automatically.<\/p>\n
      Configure Webhook For Multibranch Pipeline<\/h2>\n
      \n
      Important Note:<\/strong><\/b> If you have a GitHub App created for Jenkins with webhook configuration and installed in the repositories, you don’t need to configure the webhook separately.<\/div>\n<\/div>\n
      Follow the steps given below to setup the Jenkins webhook on the repo.<\/p>\n
      Step 1: <\/strong>Head over to the Github repo and click on the settings.<\/p>\n
      \"Github<\/figure>\n
      Step 2: <\/strong>Select the webhook option at the left and click \u201cAdd Webhook\u201d button.<\/p>\n
      \"Github<\/figure>\n
      Step 3: <\/strong>dd your Jenkins URL followed by \u201c\/github-webhook\/<\/code><\/strong>\u201d under payload URL. Select the content type as \u201capplication\/json\u201d and click \u201cAdd Webhook\u201d<\/p>\n
      \n
      Note: <\/strong><\/b>You can choose what type of webhook you want to receive in Jenkins. For example, you want to trigger the pipeline only during PR; then, you can select just the PR event from the \u201cLet me select individual events\u201d option.<\/div>\n<\/div>\n
      \"\"<\/figure>\n
      You should see a green tick mark on a successful webhook configuration as shown below.<\/p>\n
      \"Jenkins<\/figure>\n
      If you don\u2019t see a green tick or see a warning sign, click on the webhook link, scroll down to \u201cRecent Deliveries,\u201d and click on the last webhook. You should be able to view why the webhook delivery failed with the status code.<\/p>\n
      \"troubleshoot<\/figure>\n
      Now we are done with all the required configurations for the multi-branch pipeline. The next step is to test the multi-branch pipeline workflow triggers.<\/p>\n
      Test Multi-branch Pipeline<\/h2>\n
      This repo am using has two branches. main<\/code> and develop<\/code>.<\/p>\n
      Update some content in the README file in the develop branch and raise a PR to main. You can also do this from a forked repository.<\/p>\n
      It will send a webhook to Jenkins and and it will create a pipeline for the raised PR and starts building it.<\/p>\n
      Now, if you check Jenkins you will find a pipeline for PR in Jenkins as shown below.<\/p>\n
      \"\"<\/figure>\n
      If the build fails, you can commit the changes to the develop branch and as long as the PR is open, it will trigger the PR pipeline.<\/p>\n
      In the Jenkinfile<\/code><\/strong> I have added a condition to skip the deploy stage if the branch is develop and main if it is  a PR build. You can check that in the Jenkins build stages. If you check the stages you can clearly see the skipped deployment stages as shown below.<\/p>\n
      \"\"<\/figure>\n
      Now if you merge the PR, Jenkins will trigger the main branch pipelin<\/strong> and executes all the deployment stages mentioned in the Jenkinsfile for the main branch.<\/p>\n
      For example, in out pipeline example, we skip the dev and QA deployment if the branch is main. You can see that in the stages output.<\/p>\n
      \"\"<\/figure>\n
      Enabling Pull Request Status Checks<\/h2>\n
      With multi-branch pipeline, you can enable status checks. Meaning Jenkins reports the results back to GitHub as status checks. It appears on the PR page as shown below.<\/p>\n
      \"Jenkins<\/figure>\n
      For example, developers may need status checks for the following.<\/p>\n
      Common status checks might include:<\/p>\n
        \n
      1. Build verification<\/li>\n
      2. Unit tests<\/li>\n
      3. Code style checks<\/li>\n
      4. Security scans<\/li>\n<\/ol>\n
        To enable these status checks in Github PR, you need to create create a Github App for Jenkins.<\/p>\n
        You can follow this Github App for Jenkins<\/a> detailed guide to create one.<\/p>\n
        Then in the multi-branch pipeline configuration, you need to use the Github App credential instead of username and password.<\/p>\n
        \"\"<\/figure>\n
        Troubleshooting Multibranch Pipelines<\/h2>\n
        I will talk about a few possible errors in a multibranch pipeline that you might encounter and how to troubleshoot them.<\/p>\n
        Branch Discovery Issue<\/strong>  <\/p>\n
        Sometimes even after creating new branches in the SCM, it might not reflect in the Jenkins pipeline. You can try running the “Scan Repository Now” option to scan the repo again. Also, check the repository scan configurations in the pipeline.<\/p>\n
        PR Webhooks Not Triggering the Pipelines<\/strong><\/p>\n
        \u200bWhen a webhook is not triggering the pipeline, check the webhook delivery in Github for status code and error. Also, check if the Jenkins URL is correct. Also, check Jenkins logs from Manage Jenkins --> System Logs --> All Jenkins logs<\/code>. If Jenkins is able to receive the webhook, the log should show the reason why the jobs are not getting triggered.<\/p>\n
        Commits Not Triggering Pipeline<\/strong><\/p>\n
        If you want each commit to rigger the branch pipeline, then you should select the “Discover All Branches<\/strong> or ” option in the branch discovery configuration. So whenever you commit a change to the discoverable branches or raise a PR, the pipeline will automatically get triggered.<\/p>\n
        Multibranch Pipeline Best Practices<\/h2>\n
        Let’s have a look at some of the best practices for a multibranch pipeline.<\/p>\n
        Repo Branching <\/strong>\u2013 Have a Standard Structure<\/p>\n
        It is essential to have the standard branching structure for your repositories. Whether it is your application code or infra code, having a standard branching will reduce the inconsistent configurations across different pipelines.<\/p>\n
        Shared Libraries<\/strong> \u2013 Reusable Pipeline Code<\/p>\n
        Make use of shared libraries<\/a> for all your multi-branch pipelines. Reusable libraries make it easy to manage all the pipeline stages in a single place. <\/p>\n
        Pull Request Vs Commit Triggers<\/strong><\/p>\n
        Try to use a PR based pipeline rather than commit based. If a code repo gets continuous commits it might overwhelm Jenkins with many builds.<\/p>\n
        Commit based triggers are supported in PR based discovery as well. Here the commit trigger happens only when the PR is still open.<\/p>\n
        Jenkins Pipeline Vs. Multibranch Pipeline<\/h2>\n
        A normal pipeline job is meant for building a single branch from the SCM and deploy to a single environment. However, you can<\/p>\n
        A multibranch pipeline is meant for building multiple branches from a repository and deploy to multiple environments if required.<\/p>\n
        A pipeline job supports both pipeline steps to be added in Jenkins configuration and form SCM.<\/p>\n
        Use pipeline job for adhoc jobs, parameterised job executions and to debug pipeline as code.<\/p>\n
        Do not use multibranch pipeline if you do not have a standard branching and CI\/CD strategy.<\/p>\n
        Multibranch Pipeline Vs. Github Organization Job<\/h2>\n
        Like a multi-branch pipeline, the Github organization folder is one of the Jenkins project types.<\/p>\n
        \n
        Note: <\/strong><\/b>The organization folder is not just limited to Github. It can be use used for Gitlab<\/a>, Bitbucket teams<\/a>, or Gitea<\/a> organization.<\/div>\n<\/div>\n
        Mullitbranch pipleine can only configure pipelines for a single Git repository. Whereas a Jenins Github organization project can automatically configure multi-branch pipelines for all the repos in a Github organization.<\/p>\n
        It can discover all the repositories in the configured Github organization, with a Jenkinsfile.<\/p>\n
        Also, you can configure a generic webhook in the organizational level to avoid having webhooks in each repo.<\/p>\n
        The only difference between multi-branch and organization project is that organizations can configure multi-branch pipelines for multiple repos.<\/p>\n
        So which one should I use?<\/p>\n
        This totally depends on the workflow you need. If you have a standard pipeline and process of deploying applications or infra code, Github organization is great. Or else, configuring multibranch pipeline separately will be a good option.<\/p>\n
        I\u2019d like to hear from you<\/h2>\n
        That\u2019s all for my guide to multi-branch pipelines.<\/p>\n
        What type of Jenkins pipleines are you using?<\/p>\n
        Do you think multi-branch pipeline will add value to your workflows?<\/p>\n
        Or maybe you want stick to regular Jenkinsfile pipelines<\/p>\n
        Either way, let me know by leaving a comment.<\/p>\n
        \n
        Ngu\u1ed3n:<\/strong> Jenkins Multibranch Pipeline Tutorial For Beginners \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
        Source: https:\/\/devopscube.com\/jenkins-multibranch-pipeline-tutorial\/<\/p>\n","protected":false},"author":1,"featured_media":885,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-884","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/884","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=884"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/884\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/885"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=884"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=884"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=884"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}