How I Structured my GitLab for a Scalable GCP + Kubernetes Setup including FluxCD
Problem Statement
Although selected because the company where I work was using it, GitLab has now been my preferred tool for my personal software project to manage my code, continuously build the various artefacts my projects are producing and deliver them continuously to my Kubernetes clusters. One of the main question I had originally was the structure of GitLab groups (which I view as folders) and projects (essentially Git repositories) so the rest of the setup is easy, simple to maintain yet following most of, if not all, the best practices so it can easily integrate with technologies I would adopt along the way.
Requirements
I had few but important requirements when I set out to build my personal project on GitLab.
Differentiation between Infrastructure and Application
My initial focus was on learning infrastructure platform engineering—technologies such as Kubernetes, Istio, Keycloak and OAuth2-proxy, I wanted to differentiate between the git repo containing the configuration for each component and the examples for each application I was building to test out my platform. Along the way, I realized and learned that it had the added benefits to provide an easy setup for various aspects of the entire codebase. RBAC could be done to set up access with limited realm if needed (more on that below in RBAC). Since GitLab provides the ability to create a default pipeline via the gitlab-ci.yml file, I was able to provide a different execution script for each (more on in Various Technologies). And finally, since I was learning pretty much everything and starting from scratch, I could reuse templates and examples provided within each of my project with minimal changes or complicated mix up within a single project.
GCP and Kubernetes
My company has used AWS and Azure extensively and is doing quite well using both. Personally however, I've always found AWS very convoluted to use and Azure still seem to have this Microsoft-based technologies first view, easily integrating Microsoft technologies but problematic on cheaper, open source solution the other cloud providers include (although that has started to change lately). I then decided to use the Google Cloud Platform for my personal projet and needed a setup that would support it easily. Over the months, I have found it very simple to use (i.e. a much better UI experience than AWS) and very stable (a few incidents but rarely on product I use!).
RBAC
I have obviously very limited need in that area for my personal project with a team of one: me! However, since I was using this project to learn various aspects of platform engineering, I was also trying to take in some requirements from my company so I can better assess a good way to set it up. As such, one of the big areas was access segmentation and ensuring security was built in with a zero-trust network and RBAC system easy to manage.
Flux CD
Since I was using GitLab and GCP, I always knew I would eventually get to a proper GitOps setup on my system. Although I didn't know at the time FluxCD would end up to be my choice, I wanted to ensure I could integrate easily. GitLab uses and recommends FluxCD as the default GitOps choice tool. And since FluxCD uses standard command line functions similar to "kubectl" or "istioctl", it was my preferred choice.
Various Technologies
One aspect I didn't realize originally but in all honesty got lucky with my setup was the fact that having a repo for each component is making it much easier to configure and maintain. Each component has its own technologies; each component has its own timing when to continuously integrate or continuously deliver; each component requires a specific pipeline that by virtue of separating each component, can be focused solely on that component.
Groups & Projects Structure
The folder structure is quite simple, yet it has allowed me to maintain a simple codebase for all of my components and ensure I can expand it along the way with new technologies or system added over time.
Setup Explanation: Structure
Bouc.io
├── Application // All examples I used to test my infrastructure components and eventually my chatbot app.
│ └── Examples // Each will have 2 git repos, one for the app code itself, the other for the Helm chart code.
│ ├── api-example // A simple NodeJs API server.
│ │ ├── api-chart
│ │ └── api-example
│ ├── api-springboot-example // A simple Java Springboot API server.
│ │ ├── api-java-chart
│ │ └── api-java-example
│ ├── chatbot-example // A simple Chatbot app with the API server, the UI, their respective Helm charts and the Ollama chart.
│ │ ├── chatbot-api
│ │ ├── chatbot-api-chart
│ │ ├── chatbot-chart
│ │ ├── chatbot-ui
│ │ └── ollama-chart
│ ├── static-website // A simple static website built over Nginx.
│ │ ├── static-web-chart
│ │ └── static-web-example
│ └── txn-web-example // A simple REACT Singlepage app.
│ ├── txn-web-chart
│ └── txn-web-example
├── Documentation
│ ├── Getting Started // Documentation to complete the setup locally.
│ └── Postman Setup // The Postman configuration locally (although replaced when in teams with the shared workspace).
├── FluxCD
│ └── fluxcdboucio // My FluxCD git repo.
├── Infrastructure
│ ├── GCP // My GCP configuration for my network, cluster and bastion hosts along with the required databases.
│ │ ├── GCP-Bastion-Host
│ │ ├── GCP-Cluster-Network
│ │ ├── GCP-Keycloak-DB
│ │ ├── GCP-OAuth2-Proxy-DB
│ │ ├── GKE-Cluster
│ │ ├── martincoteca-static
│ │ └── terraform-pipeline-template
│ ├── Cert-Manager
│ ├── External-DNS
│ ├── Grafana
│ ├── Istio
│ ├── k8s-dashboard
│ ├── Keycloak
│ ├── Metrics-Server
│ ├── OAuth2-Proxy
│ └── Prometheus
A Few Lessons Learned...
Still new to the process and there are far better engineers than me on pretty much every single component I am using or leveraging, I have learned a few lessons that could benefit others.
- First, if a specific setup or configuration is always "getting the way", making your life as a software engineer harder, then you probably have the wrong setup or configuration. Each time I was added new technologies, added new workflows and simply finally ending up using the full breadth of the technologies and tools at my disposal, the setup shared was easy to use and adjust. The one time it wasn't (a GCP project configuration with Terraform and git repos for each - that'll be another blog post), I ended up changing the structure to suit my needs and got a lot easier to leverage each. Each added piece was fitting nicely with the rest, allowing me to quickly add functionalities or workflows with very little problems reusing examples provided the documentation.
- Second, it is very important to properly understand a technology when starting to use it. This might sound obvious but too many times, I've seen especially often in my work life, a tool is used or elected to be used because it is the new "new thing" or "an architect has mandated it" without a proper understanding of the tool itself, the needs of the project and the impacts further down such as maintenance, security, etc. Trying to take time to identify the proper requirements, validate the tool through its use (e.g. a simple POC) and simulating a full lifecycle (i.e. installation -> configuration -> upgrade -> maintenance, etc.) can save a lot of time and headaches later on.
Conclusion
This GitLab structure, though imperfect and still evolving, has allowed me to maintain a modular, secure, and scalable foundation for my personal GCP-hosted Kubernetes environment. Its separation of infrastructure and application concerns simplifies onboarding new tools and has been ensuring so far future extensibility while respecting each tool's best practices.
Share your experience if you are using it or a similar approach.
AI Usage Disclosure
This document was created with assistance from AI tools. The content has been reviewed and edited by a human. For more information on the extent and nature of AI usage, please contact the author.