Complex systems.
Clear thinking.

Background

When I joined the team, our monorepo had around 20 components. Today, we manage nearly 200, making static configurations for CI pipelines, Docker images, Kubernetes manifests, and Docker Compose files nearly impossible to handle without errors, exceptions, and hacks.

The Challenge

Manually maintaining these configs led to inconsistencies and surprises in deployments. Static files couldn’t scale with our growth, creating maintenance nightmares across the repository.

Our CUE Solution

We adopted CUE, a language designed exactly for this—templating, validating, and generating configurations. Our team built a custom framework that auto-generates CI scripts, Dockerfiles, Docker Compose, Semgrep rules, and more for every component from shared templates.

Key Benefits

• Taxonomy of components dropped sharply—no more custom hacks per service.
• Template changes propagate instantly across the entire repo, with immediate visibility of impacts.
• Everything is testable upfront, eliminating deployment surprises .
• All generated configs are fully standalone; I can easily extract a single component to its own repository where it runs independently.
• Or manually trigger CI locally if the main system goes down.

This approach brought real order to our monorepository, and I love how it scales with our team’s ambitions while giving us true flexibility.

Project overview

When I joined the development team, our research colleagues needed a reliable way to standardize their personal and shared virtual machine configurations. What started as a solution for just 5 researchers has scaled to support 30 researchers, 15 developers plus a growing analytics team. This project became particularly valuable with the shift to Apple M1 laptops, where x86_64 emulation caused brutal performance drops for some local components.

Technical Solution

The system uses Terraform to provision VMs in OpenStack, handling infrastructure creation declaratively and scalably. Ansible then takes over for fine-grained OS configuration, ensuring consistency across machines. Together, they manage more than 50 VMs under my control — personal ones, shared ones, and a few equipped with cutting-edge GPUs. With minor adaptations, the same approach now configures my Kubernetes clusters too.

Key Benefits

Remote terminal access to these VMs proved essential for colleagues traveling, letting them connect without running heavy local emulations and saving precious mobile data. The automation eliminated manual setup errors and adapted smoothly as our team grew sixfold. Personally, building this gave me hands-on mastery of IaC tools while directly boosting team productivity.

Background

At the beginning of our recommendation system’s development, we used simple algorithms focused on popularity, trending topics, and readership metrics. These early methods were easy to implement and produced acceptable results. However, as the user base and content volume grew, it became clear that automated content recommendations were far more effective in engaging users than editorial selection. This realization led us toward building a data-driven recommendation system powered by modern machine learning techniques.

Challenge

The initial architecture relied on manually configured models, trained and deployed by developers. Every adjustment to the training setup required rebuilding Docker images and creating a new deployment release. This process slowed experimentation and limited iteration speed.

As we added more complex models using Vowpal Wabbit and later TensorFlow, new challenges emerged:

• The need for systematic A/B testing to evaluate model performance.
• High resource costs from idle trainers waiting for manual configuration.
• Inefficient memory usage caused by the serving layer loading all trained models into a single production server.

These architectural limitations restricted both scalability and experimentation freedom.

Solution

To overcome these challenges, we introduced several technological improvements that gradually evolved into a comprehensive ML Ops framework.

1. Consul-Based Configuration We redefined the training process by introducing multiple trainers with dynamic configurations stored in Consul. This allowed non-technical team members to configure experiments directly without developer involvement. Result: Faster testing cycles and easier experiment management.

2. AdminWeb Platform We developed AdminWeb, a user-facing interface for product and research teams. It centralized experiment definitions, making configuration management accessible and transparent. Result: Unified control and visibility of experiments.

3. Model Manager and Terraform Integration The Model Manager retrieved configuration data from AdminWeb and used Terraform to provision all necessary Kubernetes resources — including trainers, serving deployments, services, and configuration maps. Result: Fully automated lifecycle management. When an experiment was removed, Terraform automatically cleaned up all related resources, ensuring a consistent and efficient infrastructure.

Outcome

This transformation ultimately shaped a functional and scalable ML Ops pipeline for our recommendation system. It allowed:

• Rapid experimentation and controlled A/B testing.
• Reduced operational overhead and better resource utilization.
• Seamless integration between machine learning workflows and production infrastructure.

The result was a more stable, efficient, and transparent system for delivering personalized recommendations — a process that evolved from simple heuristics into a robust ML-driven operation.

Project overview

Self-service deployment into production marked a key evolution in our team’s operations. As a former admin, I witnessed firsthand how manual processes created bottlenecks, but Kubernetes and GitOps tools like ArgoCD empowered developers to take ownership.

From Admin Dependency to Autonomy

Back when I managed deployments, every production release required admins to manually move Debian packages to the right branch, select servers with enough capacity, and deploy components. Kubernetes eliminated this routine work, yet admins still controlled production access. With our internal cloud maturing and admin teams specializing, developers gained independent deployment rights—shifting mindsets from gatekeeping to collaboration.

CI/CD Pipeline Evolution

Initially, we adapted our CI/CD with an internal tool that generated Kubernetes manifests for web-based deployment. Today, ArgoCD handles this seamlessly via GitOps, syncing changes declaratively and reducing errors. This upgrade shortened deployment times and reduced manual work.

Developer Ownership and On-Call Impact

Our team now holds production on-call because no one understands our product better than its builders. Reliable production quality means each weekly rotator gets financial compensation plus a personal day off—a fair win that boosts motivation. This autonomy fosters faster feedback and accountability, aligning with industry gains in productivity and reduced bottlenecks.

Project Overview

Migrating our components to Kubernetes marked a pivotal shift in our deployment strategy, streamlining operations and boosting efficiency.

Packaging Shift

We transitioned library packaging from Debian packages to Python packages, simplifying dependencies and improving portability across environments. Component packaging evolved from Debian builds to Docker images, enabling consistent runtime behavior regardless of the host system.

Deployment Automation

Manual node selection by admins for sufficient resources gave way to Kubernetes’ automated scheduling, eliminating human error and downtime. This change ensured optimal resource allocation dynamically.

Configuration Evolution

Fixed configuration files were replaced with environment variables, offering greater flexibility for different environments like dev, staging, and production. Updates became simpler without rebuilding entire packages.

Key Benefits

Deployment speed surged, with faster rollouts and rollbacks thanks to Kubernetes’ orchestration . Resource hunting time dropped to zero, as the cluster handles provisioning automatically, allowing our admin team to focus on other tasks rather than legacy ops. This migration not only cut deployment times but also infused scalability into our portfolio.

Project Overview

The primary task was to migrate processes and data from HP OpenView Service Desk to the new HP Service Manager software for Czech Police. My role involved leading the design and the migration, ensuring minimal disruption to police operations. I completed the project on time, delivering a fully functional system despite tight delivery schedule.

Key Challenges

The biggest hurdle was not data migration itself, but recreating a similar look & feel in the new UI to ease user transition. Customer had to accept small differences, which we addressed through targeted training sessions. This required close collaboration with end-users to map old workflows intuitively.

Solution and Process

We started with data export/import, validating every record for accuracy in a secure environment. For UI alignment, I designed custom dashboards and workflows in Service Manager, iterating based on feedback from police IT staff. Prototypes were tested in phases, with hands-on demos to build confidence.

Results and Learnings

The project was successfully delivered, with users reporting high satisfaction and quick adoption. It boosted efficiency in ticket handling by streamlining processes. Personally, this was a pleasant experience—it showed me how empathy in change management turns potential resistance into smooth success.

The project

In summer 2013, Vysocina region tasked us with deploying 47 interactive kiosks across museums and galleries to showcase digitized exhibits on https://mgvysociny.cz . Beyond hardware, I designed a monitored network with remote management, self-restarts, content lockdown, and The Dude integration—turning remote sites into reliable digital hubs.

Key Challenges

Museums often lacked technically skilled personnel. Kiosks needed auto-recovery and VPN-secured access to the central management site. Hurdles included automating RouterOS scripting, building OpenVPN certificate hierarchies, and training non-technical colleagues on static LAN/WiFi setups.

Software & Hardware Stack

Windows PC with SiteKiosk provided terminal lockdown and central Server console for configs/updates. Each terminal featured MikroTik RouterBOARD as OpenVPN client, plus a device for temp/humidity monitoring and remote power cycling. Kiosks acted as LAN clients.

Results & Lessons

All 47 kiosks went live in summer 2013. Automation cut deploy time and mitigated configuration errors. Key takeaway: script early for edge infra. This honed my DevOps skills for remote resilience.