{"id":950,"date":"2024-07-23T01:49:00","date_gmt":"2024-07-23T01:49:00","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=950"},"modified":"2024-07-23T01:49:00","modified_gmt":"2024-07-23T01:49:00","slug":"jenkins-build-agents-kubernetes","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=950","title":{"rendered":"How to Setup Jenkins Build Agents on Kubernetes Pods"},"content":{"rendered":"

In this Jenkins tutorial, I explained the detailed steps to set up Jenkins controller and scale Jenkins build agents on Kubernetes pods using the Jenkins Kubernetes plugin<\/p>\n

In another post, I explained how to set up Docker-based Jenkins agents<\/a>.<\/p>\n

If you have a Kubernetes cluster<\/a> in your environment, running Jenkins agents on the Kubernetes pods will give you good build isolation for different application versions.<\/p>\n

Also, an ephemeral Kubernetes pod-based Jenkins agent is a great way to reduce the cost of the CI environment, as Jenkins agents are spun up only if there is a build request.<\/p>\n

How Do Jenkins Kubernetes Pod Agents Work?<\/h2>\n

Before getting into the implementation, let’s understand how this setup works.<\/p>\n

The following image shows the high-level workflow.<\/p>\n

\"\"
Click to view in HD<\/span><\/figcaption><\/figure>\n
    \n
  1. Whenever you trigger a Jenkins<\/a> job, the Jenkins Kubernetes plugin will make an API call to create a Kubernetes agent pod.<\/li>\n
  2. Then, the Jenkins agent <\/a>pod is deployed in Kubernetes with a few environment variables containing the Jenkins server details and secrets.<\/li>\n
  3. When the agent pod comes up, it uses the details in its environment variables<\/strong> and talks back to Jenkins using the JNLP method. The following images show the environment variables of the agent pod.<\/li>\n<\/ol>\n
    \"Jenkins<\/figure>\n

    All the build steps from the Jenkinsfile run on that pod. Once the build is complete, the pod will be terminated automatically. However, there are also options to retain the build pod.<\/p>\n

    The Jenkins Kubernetes plugin handles all the communication from Jenkins to the Kubernetes cluster.<\/p>\n

    Also, as long as your Kubernetes cluster scales, you can scale your Jenkins build agents without issues.<\/p>\n

    Setting Up Jenkins Build Pods On Kubernetes<\/h2>\n

    To work on this setup, we need the following.<\/p>\n

      \n
    1. A working Kubernetes cluster.<\/li>\n
    2. Kubernetes admin user to create Kubernetes deployments<\/a> and service accounts<\/li>\n
    3. A running Jenkins controller<\/li>\n<\/ol>\n

      Also, I am considering two scenarios here.<\/p>\n

        \n
      1. Jenkins controller running inside the Kubernetes cluster.<\/li>\n
      2. Jenkins controller running outside the Kubernetes cluster.<\/li>\n<\/ol>\n

        We will look at both scenarios and their configurations.<\/p>\n

        Overfall, here is what we are going to do.<\/p>\n

          \n
        1. Create a namespace devops-tools<\/code><\/li>\n
        2. Create a Kubernetes service account named jenkins-admin<\/code> with permissions to manage pods in devops-tools<\/code> namespace. Jenkins will use this service account to deploy the agent pods. (Both internal & external Jenkins)<\/li>\n
        3. Deploy Jenkins in devops-tools <\/code>namespace with the jenkins-admin<\/code> service account. (If you don’t have an existing Jenkins)<\/li>\n
        4. Configure the Kubernetes Jenkins Plugin so that Jenkins can interact with the Kubernetes cluster and deploy build agents.<\/li>\n<\/ol>\n

          Setting Up Kubernetes Namespace & Service Account<\/h2>\n

          Let’s get started with the setup.<\/p>\n

          Step 1: <\/strong>Create a namespace called devops-tools<\/code><\/p>\n

          kubectl create namespace devops-tools<\/code><\/pre>\n
          Step 2: <\/strong>Save the following manifest as service-account.yaml<\/code>. It contains the role and role-binding for the service account with all the permission to manage pods in the devops-tools<\/code> namespace.<\/p>\n
          apiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: jenkins-admin\n  namespace: devops-tools\n---\napiVersion: rbac.authorization.k8s.io\/v1\nkind: Role\nmetadata:\n  name: jenkins\n  namespace: devops-tools\n  labels:\n    \"app.kubernetes.io\/name\": 'jenkins'\nrules:\n- apiGroups: [\"\"]\n  resources: [\"pods\"]\n  verbs: [\"create\",\"delete\",\"get\",\"list\",\"patch\",\"update\",\"watch\"]\n- apiGroups: [\"\"]\n  resources: [\"pods\/exec\"]\n  verbs: [\"create\",\"delete\",\"get\",\"list\",\"patch\",\"update\",\"watch\"]\n- apiGroups: [\"\"]\n  resources: [\"pods\/log\"]\n  verbs: [\"get\",\"list\",\"watch\"]\n- apiGroups: [\"\"]\n  resources: [\"secrets\"]\n  verbs: [\"get\"]\n---\napiVersion: rbac.authorization.k8s.io\/v1\nkind: RoleBinding\nmetadata:\n  name: jenkins-role-binding\n  namespace: devops-tools\nroleRef:\n  apiGroup: rbac.authorization.k8s.io\n  kind: Role\n  name: jenkins\nsubjects:\n- kind: ServiceAccount\n  name: jenkins-admin\n  namespace: devops-tools<\/code><\/pre>\n
          Create the service account.<\/p>\n
          kubectl apply -f service-account.yaml<\/code><\/pre>\n
          The next step is to create a Secret, which creates a token for the service account.<\/p>\n
          Create a YAML file secret.yaml<\/strong> and copy the below contents into it<\/p>\n
          apiVersion: v1\nkind: Secret\nmetadata:\n  name: sa-token-secret\n  namespace: devops-tools\n  annotations:\n    kubernetes.io\/service-account.name: jenkins-admin\ntype: kubernetes.io\/service-account-token<\/code><\/pre>\n
          Create the secret<\/p>\n
          kubectl apply -f secret.yaml<\/code><\/pre>\n
          This creates a secret and links it with the service account using the annotation.<\/p>\n
          Jenkins Controller Setup in Kubernetes<\/h2>\n
          In this setup, we will have both the Jenkins controller and agents deploying in the same Kubernetes cluster.<\/p>\n
          We will set up the Jenkins controller server on the Kubernetes cluster.<\/p>\n
          \n
          Note:<\/strong><\/b> If you have an existing setup, you can use that as well. Ensure it has a service account with permissions to deploy pods in the namespace where Jenkins is deployed.<\/div>\n<\/div>\n
          Save the following manifest as deployment.yaml<\/code>. This manifest contains persistent volume claim, deployment, and service definitions.<\/p>\n
          \n
          Note:<\/strong><\/b> Ensure that your Kubernetes cluster setup supports persistent volumes. If you deploy Jenkins without a persistent volume, you will lose the Jenkins data on every restart or pod deletion.<\/div>\n<\/div>\n
          Also, if you are using a local cluster, create a persistent volume before running the below manifest.<\/p>\n
          # Persistent Volume Claim\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: jenkins-pv-claim\n  namespace: devops-tools\nspec:\n  accessModes:\n    - ReadWriteOnce\n  resources:\n    requests:\n      storage: 50Gi\n\n# Deployment Config\n---\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n  name: jenkins-deployment\n  namespace: devops-tools\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: jenkins\n  template:\n    metadata:\n      labels:\n        app: jenkins\n    spec:\n      serviceAccountName: jenkins-admin\n      securityContext:\n            fsGroup: 1000 \n            runAsUser: 1000\n      containers:\n        - name: jenkins\n          image: jenkins\/jenkins:lts\n          resources:\n            limits:\n              memory: \"2Gi\"\n              cpu: \"1000m\"\n            requests:\n              memory: \"500Mi\"\n              cpu: \"500m\"\n          ports:\n            - name: httpport\n              containerPort: 8080\n            - name: jnlpport\n              containerPort: 50000\n          livenessProbe:\n            httpGet:\n              path: \"\/login\"\n              port: 8080\n            initialDelaySeconds: 90\n            periodSeconds: 10\n            timeoutSeconds: 5\n            failureThreshold: 5\n          readinessProbe:\n            httpGet:\n              path: \"\/login\"\n              port: 8080\n            initialDelaySeconds: 60\n            periodSeconds: 10\n            timeoutSeconds: 5\n            failureThreshold: 3\n          volumeMounts:\n            - name: jenkins-data\n              mountPath: \/var\/jenkins_home         \n      volumes:\n        - name: jenkins-data\n          persistentVolumeClaim:\n              claimName: jenkins-pv-claim\n\n# Service Config\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: jenkins-service\n  namespace: devops-tools\n  annotations:\n      prometheus.io\/scrape: 'true'\n      prometheus.io\/path:   \/\n      prometheus.io\/port:   '8080'\nspec:\n  selector: \n    app: jenkins\n  type: NodePort  \n  ports:\n    - name: httpport\n      port: 8080\n      targetPort: 8080\n      nodePort: 32000\n    - name: jnlpport\n      port: 50000\n      targetPort: 50000\n  <\/code><\/pre>\n
          Create the deployment.<\/p>\n
          kubectl apply -f deployment.yaml<\/code><\/pre>\n
          After a couple of minutes, the Jenkins deployment will be up, and you will be able to access any Kubernetes node on the port 32000<\/code><\/p>\n
          Step 4: <\/strong>Access the Jenkins dashboard over the node port and unlock it using the password from the pod logs. Install the suggested plugins and create a Jenkins user.<\/p>\n
          Please follow the Jenkins on Kubernetes blog<\/a> if you have any doubts.<\/p>\n
          Jenkins Kubernetes Plugin Configuration<\/h2>\n
          Jenkins Kubernetes plugin <\/a>is required to set up Kubernetes-based build agents. Let’s configure the plugin.<\/p>\n
          Step 1: Install Jenkins Kubernetes Plugin<\/h3>\n
          Go to Manage Jenkins <\/code>\u2013> Manage Plugins<\/code>, search for the Kubernetes Plugin in the available tab, and install it. The following Gif video shows the plugin installation process.<\/p>\n
          \"installing<\/figure>\n
          Step 2: <\/strong>Create a Kubernetes Cloud Configuration<\/h3>\n
          Once installed, go to Manage Jenkins<\/code> \u2013> Clouds<\/code><\/p>\n
          \"configure<\/figure>\n
          Click New Cloud<\/p>\n
          \"add<\/figure>\n
          Give a name and select Kubernetes.<\/p>\n
          \"create<\/figure>\n
          Click Create and move on to the next step<\/p>\n
          Step 3: Configure Jenkins Kubernetes Cloud<\/h3>\n
          Here we have two scenarios.<\/p>\n
            \n
          1. Jenkins server running inside<\/strong> the same Kubernetes cluster<\/li>\n
          2. The Jenkins server is running outside<\/strong> the Kubernetes cluster.<\/li>\n<\/ol>\n
            Let’s look at configurations for both scenarios.<\/p>\n
            Jenkins server running inside the same Kubernetes cluster<\/h4>\n
            Since we have Jenkins inside the Kubernetes cluster with a service account to deploy the agent pods, we don’t have to mention the Kubernetes URL or certificate key.<\/p>\n
            However, to validate the connection using the service account, use the Test Connection button as shown below. It should show a connected message if the Jenkins pod can connect to the Kubernetes API Server.<\/p>\n
            \"Test<\/figure>\n
            Jenkins server running Outside the Kubernetes cluster<\/h4>\n
            If your Jenkins server is running outside the Kubernetes cluster, you need to specify the following.<\/p>\n
              \n
            1. Kubernetes URL:<\/strong> This is the Kubernetes API Server endpoint. If it is https enabled, use the https URL.<\/li>\n
            2. Kubernetes<\/strong> Server Certificate key:  If you have a Kubernetes Cluster CA certificate, you can add it for secure connectivity. You can get the certificate from the pod location \/var\/run\/secrets\/kubernetes.io\/serviceaccount\/ca.crt<\/code> .If you do not have the certificate, you can enable the  “disable https certificate check<\/code>” option.<\/li>\n
            3. Credentials:<\/strong> For Jenkins to communicate with the Kubernetes cluster, we need a service account token with permission to deploy pods in the devops-tools<\/code> namespace.<\/li>\n<\/ol>\n
              \n
              Note:<\/strong><\/b> If you use managed services like GKE cluster<\/a>, you can get all the cluser details from the GKE dashboard.<\/div>\n<\/div>\n
              We have created the service account and assigned a secret token to the devops-tools<\/code> namespace. We need to get the token from the secret.<\/p>\n
              Execute the following commands to retrieve the secret name from the service account.<\/p>\n
              kubectl get secret sa-token-secret -n devops-tools -o jsonpath='{.data.token}' | base64 --decode<\/code><\/pre>\n
              Now click the Add<\/code> button under credentials and create a credential type “Secret text<\/code>“. Enter the service account token in the secret box and add other details as shown below. Finally, save the credential.<\/p>\n
              \"Add<\/figure>\n
              \"Add<\/figure>\n
              The Kubernetes cloud configuration would look like the following.<\/p>\n
              \"configure<\/figure>\n
              After filling in all the details, you can test the connection to validate the Kubernetes cluster connectivity.<\/p>\n
              Step 4: Configure the Jenkins URL Details<\/h4>\n
              For the Jenkins Controller running inside the cluster, you can use the Kubernetes cluster’s service endpoint as the Jenkins URL because agent pods can connect to the cluster via internal service DNS.<\/p>\n
              The URL is derived using the following syntax.<\/p>\n
              http:\/\/<service-name>.<namespace>.svc.cluster.local:8080<\/code><\/pre>\n
              In our case, the service DNS will be,<\/p>\n
              http:\/\/jenkins-service.devops-tools.svc.cluster.local:8080<\/code><\/pre>\n
              Also, add the POD label, which can be used to group the containers that can be used for billing or custom build dashboards.<\/p>\n
              \n
              Note:<\/strong><\/b> If the Jenkins controller is outside the Kubernetes cluster, use the Jenkins IP or DNS <\/strong><\/b>in the Jenkins URL configuration.<\/div>\n<\/div>\n
              \"Jenkins<\/figure>\n
              Step 5: Create POD and Container Template<\/h4>\n
              Next, you must add the POD template with the details, as shown in the image below. The label kubeagent will be used as an identifier to pick this pod as the build agent. Next, we must add a container template with the Docker image details.<\/p>\n
              \"POD<\/figure>\n
              The next configuration is the container template. If you don’t add a container template, the Jenkins Kubernetes plugin will use the default JNLP image from the Docker hub to spin up the agents. ie, jenkins\/inbound-agent<\/a><\/p>\n
              If you are on the corporate network and don’t have access to the Docker hub, you will have to build your own jnlp<\/code> image and override the default with the same name as shown below, assuming jenkins\/inbound-agent:latest<\/code> is the custom jnlp image.<\/p>\n
              Ensure that you remove the sleep<\/code> and 9999999<\/code> default argument from the container template.<\/p>\n
              \"overriding<\/figure>\n
              We can add multiple container templates to the POD template and use them in the pipeline. I have explained that in the next section with Jenkinsfile<\/code> examples.<\/p>\n
              This is the base minimum configuration required for the agent to work. Later in the pipeline examples, I will explain a few use cases for volumes and other options.<\/p>\n
              Now save all the configurations, and let’s test if we can build a job with a pod agent.<\/p>\n
              Step 6: <\/strong>Go to Jenkins home –> New Item and create a freestyle project.<\/p>\n
              In the job description, add the label kubeagent<\/code> as shown below. It is the label we assigned to the pod template. This way, Jenkins knows which pod template to use for the agent container.<\/p>\n
              \"validating<\/figure>\n
              Add a shell build step with an echo command to validate the job as shown below.<\/p>\n
              \"Jenkins<\/figure>\n
              Now, save the job configuration and click “Build Now”<\/p>\n
              You should see a pending agent in the job build history below.<\/p>\n
              \"docker<\/figure>\n
              You will see a successful build in a couple of minutes. If you check the logs, the executed shell will be shown.<\/p>\n
              \"successful<\/figure>\n
              Jenkinsfile With Pod Template<\/h2>\n
              Whatever we have seen till now is to understand and validate the Kubernetes Jenkins plugin setup.<\/p>\n
              When it comes to actual project pipelines, it is better to have the POD templates in the Jenkinsfile<\/code><\/p>\n
              Here is what you should know about the POD template.<\/p>\n
                \n
              1. By default, the plugin uses a JNLP container image to connect to the Jenkins server. You can override with a custom JNLP image provided you give the name jnlp<\/code> in the container template.<\/li>\n
              2. You can have multiple container templates in a single pod template. Then, each container can be used in different pipeline stages.<\/li>\n
              3. POD_LABEL<\/code> will assign a random build label to the pod when the build is triggered. You cannot give any other names other than POD_LABEL<\/code><\/li>\n<\/ol>\n
                Here is an example Jenkinsfile<\/code> with a POD template.<\/p>\n
                podTemplate {\n    node(POD_LABEL) {\n        stage('Run shell') {\n            sh 'echo hello world'\n        }\n    }\n}<\/code><\/pre>\n
                Building the above Jenkinsfile in a pipeline job will use the default JNLP image and execute the commands in the “Run Shell” stage. When I say default, the plugin will use the JNLP image from the docker hub if you don’t specify any.<\/p>\n
                Now, you can use your own jnlp<\/code> image using a containerTemplate<\/code> with all necessary build tools and use them in the pipeline as given below.<\/p>\n
                Here, instead of jenkins\/inbound-agent:latest<\/code>, you will have your own image.<\/p>\n
                podTemplate(containers: [\n    containerTemplate(\n        name: 'jnlp', \n        image: 'jenkins\/inbound-agent:latest'\n        )\n  ]) {\n\n    node(POD_LABEL) {\n        stage('Get a Maven project') {\n            container('jnlp') {\n                stage('Shell Execution') {\n                    sh '''\n                    echo \"Hello! I am executing shell\"\n                    '''\n                }\n            }\n        }\n\n    }\n}<\/code><\/pre>\n
                Multi Container Pod Template<\/h2>\n
                You can use multiple container templates in a single POD template.<\/p>\n
                Here is a use case of this setup.<\/p>\n
                Let’s say you want to set up a single build pipeline that builds both Java and Python projects. In this case, you can use two container templates in the build stages.<\/p>\n
                \"\"<\/figure>\n
                In the following example, in two separate stages, we are calling two different containers specified in the pod template.<\/p>\n
                One container contains all the maven dependencies for Java build, and another contains Python build dependencies.<\/p>\n
                podTemplate(containers: [\n    containerTemplate(\n        name: 'maven', \n        image: 'maven:3.8.1-jdk-8', \n        command: 'sleep', \n        args: '30d'\n        ),\n    containerTemplate(\n        name: 'python', \n        image: 'python:latest', \n        command: 'sleep', \n        args: '30d')\n  ]) {\n\n    node(POD_LABEL) {\n        stage('Get a Maven project') {\n            git 'https:\/\/github.com\/spring-projects\/spring-petclinic.git'\n            container('maven') {\n                stage('Build a Maven project') {\n                    sh '''\n                    echo \"maven build\"\n                    '''\n                }\n            }\n        }\n\n        stage('Get a Python Project') {\n            git url: 'https:\/\/github.com\/hashicorp\/terraform.git', branch: 'main'\n            container('python') {\n                stage('Build a Go project') {\n                    sh '''\n                    echo \"Go Build\"\n                    '''\n                }\n            }\n        }\n\n    }\n}<\/code><\/pre>\n
                You can try building the above Jenkinsfile using the pipeline job.<\/p>\n
                While building the above pipeline, if you check the Kubernetes pods, you will see three containers in the build agent pod, as shown below.<\/p>\n
                \"Kubernetes<\/figure>\n
                \n
                Note: You cannot use the Docker hub images directly due to security compliance issues in actual projects. So you have to build your own Docker images and host them in the organization-approved container registry.<\/div>\n<\/div>\n
                Using Shared Persistent Volumes With Jenkins Docker Agent Pods<\/h2>\n
                It is better to attach a shared persistent volume to the build container to speed up the build process.<\/p>\n
                For example, if you take a Java application, it has many Maven package dependencies.<\/p>\n
                When you build the Java apps, it downloads dependencies added in the pom.xml from the remote maven repository the first time, and it creates a local .m2 cache directory where the dependent packages are cached.<\/p>\n
                The .m2<\/code> cache is impossible in Docker agent-based builds as it gets destroyed after the build.<\/p>\n
                To solve this issue, we can create a persistent volume for the maven cache and attach it to the agent pod via the container template.<\/p>\n
                To demonstrate this, first, let’s create a PVC<\/p>\n
                apiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: maven-repo-storage\n  namespace: devops-tools\nspec:\n  accessModes:\n    - ReadWriteOnce\n  resources:\n    requests:\n      storage: 50Gi<\/code><\/pre>\n
                Here is an example Jenkinsfile<\/code> with a POD template that uses the maven-repo-storage<\/code> persistent volume.<\/p>\n
                podTemplate(containers: [\n  containerTemplate(\n      name: 'maven', \n      image: 'maven:latest', \n      command: 'sleep', \n      args: '99d'\n      )\n  ], \n  \n  volumes: [\n  persistentVolumeClaim(\n      mountPath: '\/root\/.m2\/repository', \n      claimName: 'maven-repo-storage', \n      readOnly: false\n      )\n  ]) \n\n{\n  node(POD_LABEL) {\n    stage('Build Petclinic Java App') {\n      git url: 'https:\/\/github.com\/spring-projects\/spring-petclinic.git', branch: 'main'\n      container('maven') {\n        sh 'mvn -B -ntp clean package -DskipTests'\n      }\n    }\n  }\n}<\/code><\/pre>\n
                Building Docker Images On Kubernetes Cluster<\/h2>\n
                If you are using Docker to deploy applications, you can integrate your CI Docker build pipeline with Kubernetes agents.<\/p>\n
                There are a few ways to run Docker on Docker<\/a> for build use cases. However, since Kubernetes removed Docker runtimes, it is better to use alternative solutions.<\/p>\n
                For now, the best way to build docker images on the Kubernetes cluster is using Kaniko<\/a><\/p>\n
                Refer building docker image using kaniko <\/a>to learn more about kaniko build pipeline using Jenkins pipeline.<\/p>\n
                Conclusion<\/h2>\n
                If you are using Jenkins & Kubernetes, you should definitely try out the container-based agents.<\/p>\n
                Scaling your Jenkins agents on Kubernetes helps you avoid the administrative overhead associated with static build VMs. Even though dynamic VM build options are available, each build could take a long time compared to dynamic container agents.<\/p>\n
                You don’t have to worry about running out of resources for Jenkins builds.<\/p>\n
                Do give it a try, and let me know if you face any issues.<\/p>\n
                Also, check out my comprehensive list of Jenkins tutorials for beginners<\/a><\/p>\n
                \n
                Ngu\u1ed3n:<\/strong> How to Setup Jenkins  Build Agents on Kubernetes Pods \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
                Source: https:\/\/devopscube.com\/jenkins-build-agents-kubernetes\/<\/p>\n","protected":false},"author":1,"featured_media":951,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-950","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/950","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=950"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/950\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/951"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=950"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=950"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=950"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}