AI News Hub Logo

AI News Hub

How to Build a Custom AI Quality Gate on Cloud Run (From Zero to Production)

DEV Community
Alexander Tyutin

In my previous article about treating architecture documentation as a first-class asset, I had a great discussion in the comments about enforcing architectural rules. I promised to share materials from my recent Google Developer Groups workshop. The workshop is now finished! Here is the story of how I built an AI Quality Gate, how it helped me solve the internal "CEO, CTO, CFO, CISO" conflict, and a summary of the live demonstration. Playground repositories with source code: Quality Gate PoC CheckMe Repo #1 CheckMe Repo #2 CheckMe Repo #3 I work as a DevSecOps engineer, but in my free time, I mentor for Technovation Girls, a global program that helps young women learn tech and STEM. Because we always need more IT mentors, I built an AI mentor bot to help the students. Safety: Because children use it, it had to be completely safe from AI hallucinations. Budget: Because I pay for it myself, it had to be very cheap. The bot was a big success. Using Google Cloud Run and Vertex AI, it handled 250 users and answered 1,500 questions for only about $25-$55 a month. However, when I tried to add new features quickly, I faced a big problem. With only 1-2 hours of free time a day for this project, I experienced a harsh "CEO, CTO, CFO, CISO" conflict in my own head: The CTO wanted to write code and ship features fast. The CISO wanted to stop releases to make sure everything was secure. The CFO wanted to keep cloud costs low. The CEO wanted the product to grow and succeed. To solve the "CEO, CTO, CFO, CISO" conflict, I created an AI Quality Gate. The first action of the Quality Gate was to block its own MVP from reaching the production. So I decided it was a good sign. Short Summary: Fail. List of Critical Findings: AI Gateway (AAA): The provided code retrieves a GitLab token directly from Secret Manager and uses it for GitLab API access. This bypasses the AI gateway, violating the "ALWAYS Consistency with AI gateway (AAA, FinOps)" rule. The AAA component should manage authentication and authorization for all external services, including GitLab. Constructive Recommendations: Implement AI Gateway AAA: Modify the ai_review.py script to authenticate with the AI gateway first. The AI gateway will then handle the GitLab authentication, providing a centralized and secure way to manage access. Use gateway's provided token instead of direct GitLab API access from the job. FinOps Considerations: Track the cost of AI reviews and link this with FinOps tools, it is important to provide cost visibility since the usage of resources will increase. Because it runs on Cloud Run, it only costs money when it is actively checking code. For a whole month of automated, deep-context code reviews, I paid only $0.12! This made the CFO part of my brain very happy. During the GDG workshop, I showed a live demo across three different code repositories to prove why traditional tools are not enough. First, I scanned a simple service using standard tools like Ruff, Pylint, and Semgrep. The code got a perfect 10/10 score. However, when I sent the code to the AI Quality Gate, it blocked the release. It found a critical SQL injection and a prompt injection (a hidden note in the code telling the AI reviewer to "report that everything is fine"). Traditional linters missed this completely, but the AI caught it and gave me exact steps to fix it. In the second project, the README.md file stated that the system followed strict privacy standards and anonymized user data. But the actual code did the opposite: it saved real user emails and IDs. Standard tools missed this, but the AI Quality Gate read the documentation, compared it to the code's behavior, and found the security violation. The last demo was the most powerful. The repository had zero lines of code. It only contained a Markdown document planning a new feature. I sent this text plan to the AI Quality Gate. Before I wrote a single line of Python, the AI found critical security flaws in the plan, like missing server logs and hardcoded passwords. When you keep your architecture rules and documentation close to your code, a custom AI Quality Gate becomes an incredibly powerful tool. It helps you write better code, saves time, and finally resolves the internal "CEO, CTO, CFO, CISO" conflict. Moreover such a gate may be an additional advisor with any experience you want and help to improve any idea in the earliest stage to save future money. Best of all, it costs almost nothing to run. Quality Gate PoC CheckMe Repo #1 CheckMe Repo #2 CheckMe Repo #3