{"id":496,"date":"2025-09-08T10:57:04","date_gmt":"2025-09-08T10:57:04","guid":{"rendered":"https:\/\/blog.ngocha.biz\/?p=496"},"modified":"2025-09-08T10:57:04","modified_gmt":"2025-09-08T10:57:04","slug":"observability-as-code-with-grafana-git-sync","status":"publish","type":"post","link":"https:\/\/blog.ngocha.biz\/?p=496","title":{"rendered":"Observability as Code with Grafana Git Sync"},"content":{"rendered":"<p>Infrastructure and application code have version control, but <a href=\"https:\/\/devopscube.com\/integrate-visualize-prometheus-grafana\/\" rel=\"noreferrer\">Grafana dashboards <\/a>often don\u2019t. They get edited directly in production without proper tracking. <\/p>\n<p>Git Sync in <a href=\"https:\/\/grafana.com\/docs\/grafana\/latest\/observability-as-code\/provision-resources\/intro-git-sync\/?ref=devopscube.com\" rel=\"noreferrer\">Grafana v12<\/a> (experimental) enables version control for dashboards while keeping the familiar UI-based editing.<\/p>\n<p>By the end of this blog, you will,<\/p>\n<ul>\n<li>Understand why Grafana dashboards need proper version control.<\/li>\n<li>Learn how Git Sync in Grafana v12 solves this problem without losing the familiar UI.<\/li>\n<li>Set up Git Sync step by step with <a href=\"https:\/\/devopscube.com\/what-is-docker\/\" rel=\"noreferrer\">Docker<\/a> and GitHub.<\/li>\n<li>See how bidirectional sync works in practice.<\/li>\n<li>Know the current limitations and when you should and shouldn\u2019t use it.<\/li>\n<\/ul>\n<h2 id=\"the-pain-of-managing-grafana-dashboards\">The Pain of Managing Grafana Dashboards<\/h2>\n<p>If you have worked with Grafana long enough, you have probably faced this.<\/p>\n<ul>\n<li>A production dashboard suddenly looks different. <\/li>\n<li>Someone adjusted a threshold that now floods the alert channel. <\/li>\n<li>Another person deleted a panel they thought was redundant. <\/li>\n<\/ul>\n<p>The dashboard that took weeks to perfect is now\u2026 different. Nobody knows who changed what, when, or why.<\/p>\n<p>Traditional solutions sucked.  Here is why.<\/p>\n<ul>\n<li>If you use <a href=\"https:\/\/devopscube.com\/terraform-module-best-practices\/\" rel=\"noreferrer\">Terraform<\/a>, you get dashboards as code, so you have version control (you can track every change). But you lose the Grafana visual editor.<\/li>\n<li>If you stick with the Grafana UI editor, you keep the visual editing experience, but you dont have proper version control. You cant easily track who changed what, when, or why.<\/li>\n<li>If you use backup scripts, you can save copies of dashboards at different times, but thats only like taking snapshots. It is not true version control.<\/li>\n<\/ul>\n<p>Git Sync eliminates this false choice.<\/p>\n<p>With Git Sync, you can tread your dashboards the same as your infrastructure and application code without giving up the Grafana experience.<\/p>\n<h2 id=\"what-is-git-sync\">What is Git Sync?<\/h2>\n<p>Git Sync<strong> <\/strong>creates a <strong>bidirectional bridge<\/strong> between your Grafana instance and a GitHub repository. Meaning, if you edit a dashboard in Grafana\u2019s UI, it commits to Git. If you update the dashboard JSON in GitHub, it reflects in Grafana.<\/p>\n<figure class=\"kg-card kg-image-card\"><img decoding=\"async\" src=\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/09\/image-34.png\" class=\"kg-image\" alt=\"\" loading=\"lazy\" width=\"2000\" height=\"2246\" srcset=\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/size\/w600\/2025\/09\/image-34.png 600w, https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/size\/w1000\/2025\/09\/image-34.png 1000w, https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/size\/w1600\/2025\/09\/image-34.png 1600w, https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/09\/image-34.png 2024w\" sizes=\"auto, (min-width: 720px) 720px\"><\/figure>\n<p>The synchronisation happens automatically. Each dashboard becomes a JSON file in your repository. Your entire team can <strong>collaborate on dashboards<\/strong> just like they collaborate on code &#8211; with branches, pull requests, and reviews.<\/p>\n<figure class=\"kg-card kg-image-card\"><img decoding=\"async\" src=\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/09\/image.png\" class=\"kg-image\" alt=\"Git Sync in Grafana\" loading=\"lazy\" width=\"1920\" height=\"1080\" srcset=\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/size\/w600\/2025\/09\/image.png 600w, https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/size\/w1000\/2025\/09\/image.png 1000w, https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/size\/w1600\/2025\/09\/image.png 1600w, https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/09\/image.png 1920w\" sizes=\"auto, (min-width: 720px) 720px\"><\/figure>\n<h2 id=\"prerequisites\">Prerequisites<\/h2>\n<p>The following are the requisites to get test it practically.<\/p>\n<ul>\n<li><a href=\"https:\/\/devopscube.com\/how-to-install-and-configure-docker\/\" rel=\"noreferrer\">Docker<\/a>(<code>curl [&lt;https:\/\/get.docker.com<\/code>)<\/li>\n<li>Empty <a href=\"https:\/\/github.com\/new?ref=devopscube.com\">GitHub Repo<\/a> and a <a href=\"https:\/\/github.com\/settings\/personal-access-tokens\/new?ref=devopscube.com\">Personal Access Token<\/a><\/li>\n<\/ul>\n<p>Lets get started with the setup.<\/p>\n<h2 id=\"step-1-set-up-grafana-and-prometheus-with-docker\">Step 1: Set Up Grafana and Prometheus with Docker<\/h2>\n<p>Save this as <code>docker-compose.yml<\/code><\/p>\n<pre><code class=\"language-YAML\">services:\n  grafana:\n    image: grafana\/grafana:12.2.0-16979757807 # Nightly Build\n    ports:\n      - 3000:3000\n    environment:\n      - GF_FEATURE_TOGGLES_ENABLE=provisioning,kubernetesDashboards\n  prometheus:\n    image: prom\/prometheus\n    ports:\n      - 9090:9090\n    entrypoint: \/bin\/sh\n    command: \n      - -c\n      - |\n        cat &gt; \/etc\/prometheus\/prometheus.yml &lt;&lt;EOF\n        scrape_configs:\n          - job_name: node-exporter\n            static_configs:\n              - targets: [\"node-exporter:9100\"]\n        EOF\n        prometheus --config.file=\/etc\/prometheus\/prometheus.yml\n  node-exporter:\n    image: prom\/node-exporter\n    ports:\n      - 9100:9100\n    volumes:\n      - \/proc:\/host\/proc:ro\n      - \/sys:\/host\/sys:ro\n    command: --path.procfs=\/host\/proc --path.sysfs=\/host\/sys<\/code><\/pre>\n<p>Run <code>docker compose up -d<\/code> and navigate to <code>http:\/\/localhost:3000<\/code>. <\/p>\n<p>Log in with <code>admin\/admin<\/code>. <\/p>\n<p>Add Prometheus as a Data Source: Connections \u2192 Data Sources \u2192 Add data source \u2192 Prometheus. Set URL to <code>http:\/\/prometheus:9090<\/code> and save.<\/p>\n<div class=\"kg-card kg-callout-card kg-callout-card-blue\">\n<div class=\"kg-callout-emoji\">\u26a0\ufe0f<\/div>\n<div class=\"kg-callout-text\"><i><em class=\"italic\" style=\"white-space: pre-wrap;\">Git Sync is experimental and intended for development\/test environments only.<\/em><\/i><\/div>\n<\/div>\n<h2 id=\"step-2-configure-git-sync-with-github\">Step 2: Configure Git Sync with GitHub<\/h2>\n<p>Start by creating a new GitHub repository(add a README)<\/p>\n<p>Create a GitHub <a href=\"https:\/\/github.com\/settings\/personal-access-tokens\/new?ref=devopscube.com\">Personal Access Token<\/a>:<\/p>\n<ol>\n<li>GitHub \u2192 Settings \u2192 Developer settings \u2192 Personal access tokens \u2192 <a href=\"https:\/\/github.com\/settings\/personal-access-tokens\/new?ref=devopscube.com\">Fine-grained tokens<\/a><\/li>\n<li>Select your Repository<\/li>\n<li>Generate token with these permissions:\n<ol>\n<li><strong>Contents<\/strong>: Read and Write<\/li>\n<li><strong>Metadata<\/strong>: Read<\/li>\n<li><strong>Pull requests<\/strong>: Read and Write<\/li>\n<li><strong>Webhooks:<\/strong> Read and Write<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<figure class=\"kg-card kg-image-card\"><img decoding=\"async\" src=\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/09\/image-1.png\" class=\"kg-image\" alt=\"Configuring Git Sync\" loading=\"lazy\" width=\"1920\" height=\"1080\" srcset=\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/size\/w600\/2025\/09\/image-1.png 600w, https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/size\/w1000\/2025\/09\/image-1.png 1000w, https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/size\/w1600\/2025\/09\/image-1.png 1600w, https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/09\/image-1.png 1920w\" sizes=\"auto, (min-width: 720px) 720px\"><\/figure>\n<p>In Grafana, navigate to <strong>Administration<\/strong> \u2192 <strong>Provisioning<\/strong> \u2192 <strong>GitHub<\/strong>. Fill with required data:<\/p>\n<ol>\n<li>In connect section fill\n<ol>\n<li>Personal Access Token<\/li>\n<li>Repository URL<\/li>\n<li>Branch<\/li>\n<\/ol>\n<\/li>\n<li>Choose what to synchronise section:\n<ol>\n<li>Choose: Sync all resources with external storage<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<h2 id=\"step-3-create-and-sync-your-first-dashboard\">Step 3: Create and Sync Your First Dashboard<\/h2>\n<p>Lets create dashboard and add some panels:<\/p>\n<ol>\n<li>In Dashboard Section\n<ol>\n<li>Choose Add visualisation<\/li>\n<li>Select Prometheus as Data Source<\/li>\n<\/ol>\n<\/li>\n<li>Let\u2019s Add a panels\n<ol>\n<li><strong>Memory Usage<\/strong> (Gauge): <code>(1 - (node_memory_MemAvailable_bytes \/ node_memory_MemTotal_bytes)) * 100<\/code><\/li>\n<li>Click <strong>Run queries<\/strong><\/li>\n<\/ol>\n<\/li>\n<li>Save Dashboard, Grafana commits this to GitHub\n<ol>\n<li>Add a Dashboard name in the title<\/li>\n<li>You can also name the <code>.json<\/code><\/li>\n<li>Comment will the GitHub Commit Message(<code>Add System Health dashboard with memory usage gauge<\/code>)<\/li>\n<li>You can also push to a new branch<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<figure class=\"kg-card kg-image-card\"><img decoding=\"async\" src=\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/09\/image-2.png\" class=\"kg-image\" alt=\"Synced Dashboard from Grafana UI\" loading=\"lazy\" width=\"1920\" height=\"1080\" srcset=\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/size\/w600\/2025\/09\/image-2.png 600w, https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/size\/w1000\/2025\/09\/image-2.png 1000w, https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/size\/w1600\/2025\/09\/image-2.png 1600w, https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/09\/image-2.png 1920w\" sizes=\"auto, (min-width: 720px) 720px\"><\/figure>\n<p>Check your GitHub repository. You will find a json file in Grafana folder of GitHub Repository. Your dashboard is now version controlled.<\/p>\n<h2 id=\"step-4-test-bidirectional-sync\">Step 4: Test Bidirectional Sync<\/h2>\n<p>To test bidirectional sync, we are going to edit the Memory Usage panel, switch the visualization from Gauge to Time Series, and save it with the commit message: <code>Change Gauge to Time Series<\/code><\/p>\n<div class=\"kg-card kg-callout-card kg-callout-card-blue\">\n<div class=\"kg-callout-text\"><b><strong style=\"white-space: pre-wrap;\">Heads up:<\/strong><\/b> With webhooks configured, changes sync within 5 seconds. Without webhooks, Grafana polls every 30 seconds.<\/div>\n<\/div>\n<figure class=\"kg-card kg-image-card\"><img decoding=\"async\" src=\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/09\/image-3.png\" class=\"kg-image\" alt=\"\" loading=\"lazy\" width=\"1920\" height=\"1080\" srcset=\"https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/size\/w600\/2025\/09\/image-3.png 600w, https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/size\/w1000\/2025\/09\/image-3.png 1000w, https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/size\/w1600\/2025\/09\/image-3.png 1600w, https:\/\/storage.ghost.io\/c\/5f\/2f\/5f2f4d20-2abf-4534-8d40-7aa233aedd43\/content\/images\/2025\/09\/image-3.png 1920w\" sizes=\"auto, (min-width: 720px) 720px\"><\/figure>\n<h2 id=\"step-5-fix-mistakes-using-git-history\">Step 5: Fix Mistakes Using Git History<\/h2>\n<p>Made a mistake? No problem. <\/p>\n<p>In GitHub, view the commit history for your dashboard file. Find the commit and click \u201cRevert\u201d. GitHub creates a new commit restoring the previous state. You know exactly who broke what and when.<\/p>\n<div class=\"kg-card kg-callout-card kg-callout-card-blue\">\n<div class=\"kg-callout-emoji\">\u26a0\ufe0f<\/div>\n<div class=\"kg-callout-text\">When Git Sync is enabled, GitHub becomes the source of truth. If you disconnect Git Sync, dashboards will disappear from Grafana\u2019s UI (they remain safe in Git).<\/div>\n<\/div>\n<h2 id=\"current-known-limitations\">Current known Limitations<\/h2>\n<p>Following are the current limitations of Git Sync.<\/p>\n<ol>\n<li>Only <a href=\"http:\/\/github.com\/?ref=devopscube.com\">GitHub.com<\/a> supported<\/li>\n<li>Deleted dashboards in UI don\u2019t delete from Git (by design, for safety)<\/li>\n<li>Large repositories with 200+ dashboards may experience delays<\/li>\n<li>Alert rules and data sources are not synced yet.<\/li>\n<\/ol>\n<h2 id=\"when-to-use-git-sync\">When to Use Git Sync?<\/h2>\n<p>Here is when using git Sync makes sense.<\/p>\n<ul>\n<li>You want version control but Terraform feels like overkill<\/li>\n<li>If you need audit trails (to see who changed what and when), Git Sync helps with compliance and accountability.<\/li>\n<li>When multiple engineers are editing the same dashboards, and you want to avoid conflicts or confusion.<\/li>\n<li>You need actual rollback, not Grafana\u2019s basic versioning<\/li>\n<li>Your team already does GitOps for everything else, Git Sync fits perfectly into that workflow.<\/li>\n<\/ul>\n<h2 id=\"what%E2%80%99s-next\">What\u2019s Next?<\/h2>\n<p>The goal for Git Sync is production-ready status with complete <a href=\"https:\/\/devopscube.com\/what-is-observability\/\" rel=\"noreferrer\">observability<\/a> as code capabilities. The roadmap includes syncing alert rules and data sources alongside dashboards.<\/p>\n<p>Your dashboards are as critical as your application code. They deserve the same version control, review processes, and rollback capabilities. Git Sync makes this possible without forcing you to abandon the UI that makes Grafana powerful.<\/p>\n<p>Git Sync is experimental, but it works. Connect your repo, save a dashboard, watch it appear in GitHub. Your dashboards finally get proper version control. <\/p>\n<p>When someone breaks that critical CPU panel at 3am, you will have the commit hash and the culprit\u2019s name.<\/p>\n<hr>\n<p><strong>Ngu\u1ed3n:<\/strong> <a href=\"https:\/\/devopscube.com\/observability-as-code-with-grafana-git-sync\/\" target=\"_blank\" rel=\"noopener noreferrer\">Observability as Code with Grafana Git Sync \u2014 DevOpsCube<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Source: https:\/\/devopscube.com\/observability-as-code-with-grafana-git-sync\/<\/p>\n","protected":false},"author":1,"featured_media":497,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-496","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/496","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=496"}],"version-history":[{"count":0,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/posts\/496\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=\/wp\/v2\/media\/497"}],"wp:attachment":[{"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=496"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=496"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ngocha.biz\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=496"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}