Christian Richter
Berlin, Germany+49 30 4193 6978hello@crichter.ioBook a call www.crichter.io
AI Integration (LLM, MCP) · Data Platforms · Cloud Infrastructure (AWS, GCP, Kubernetes)
Agent-ready data
I make your data ready for AI agents, correctly and safely: a semantic layer, tested query patterns, a secure MCP (Model Context Protocol) server and an accuracy test set, all in your own infrastructure. Behind it are 15 years as a data engineer, building data platforms. How it works, packages and common questions.
The same approach runs in production in Shibui Finance, my public MCP server for US stock market data:
- Market data PostgreSQL · DuckDB · 60+ years
- Semantic layer dbt models · tested SQL patterns
- MCP server read-only SQL · validated · rate-limited
- Claude questions in plain English
- 46,000
- production queries in the first six months
- 3–5%
- error rate depending on the calling model, mostly corrected by the model itself
Write-up: What 46,000 production queries taught us about LLM-generated SQL (HackerNoon). Rate limiting: caddy-jsonrpc-matcher (open source).
About
I'm an independent data and AI engineer with more than 25 years of building software, 15 of them as a data engineer, freelancing since 2010. I have delivered 25+ projects for clients in automotive, chemicals, retail, market research, media and telecommunications, taking data platforms from architecture and requirements through to production operation, and I combine hands-on engineering with technical leadership of distributed teams.
Typical engagement: design the platform, build it with the client team and hand it over.
Core stack: Python, SQL, Spark, Kafka, Airflow, dbt, Terraform, Kubernetes, AWS, GCP, MCP
References
“Christian’s expertise with distributed and AWS based infrastructures helped us big time when we needed a solution to orchestrate our servers and services, by integrating Terraform into our application lifecycle management.”
“Thanks to the use of big data technologies for our tracking backend, we are now able to analyze the behavior of our users much more precisely and to significantly improve the game play of our games.”
Selected projects
Cloud-native data platform on Kubernetes
Data Architect · Energy utility · Feb 2025 – Present
- Create architecture diagrams and documentation (backup/restore, parameterization/validation)
- Define and implement standards for Kubernetes-native services
- Document architecture decisions as ADRs
- Design and document standardized ETL pipelines
- Evaluate high-availability deployment scenarios for PostgreSQL and Kafka brokers against system requirements
- Design disaster-recovery processes for data infrastructure components
AI-assisted data extraction and ETL for an SCM platform
Data Engineer · Supply chain management · Oct 2024 – May 2026
- Helped build the master data set for a supply chain management application from 8 data sources
- Designed and implemented ~60 ETL pipelines on Google Cloud
- Designed and implemented LLM-based workflows for document data extraction and analysis (Document AI, Vertex AI), processing ~120k documents
- Analyzed and combined the data sources to improve data quality and coverage
- Built a dashboard to monitor data quality
- Set up CI/CD pipelines, monitoring and alerting for production ETL processes
Data warehouse for research and production data
Data Architect & Engineer · Chemicals · Sep 2023 – Oct 2024
- Advised on the data model, ETL pipelines and cloud architecture
- Designed and implemented multiple ETL pipelines providing access to research and production data
- Integrated SAP PLM to extract and process laboratory research data
- Integrated the manufacturing execution system (MES) for data extraction and processing
- Built an OpenSearch index for semantic search (vector search / k-NN)
- Transferred knowledge to internal staff
Migration of an on-premises data warehouse to AWS
Data Architect & Engineer · Market research · Jul 2021 – Oct 2023
- Evaluated Databricks/Delta Lake against AWS managed services/Hudi as the foundation of a data lake
- Designed and built the data warehouse on AWS Lake Formation and the Glue Data Catalog (~1.5 TB data lake)
- Migrated 100+ ETL jobs (Spark, Hive, Pig) from Cloudera to AWS Glue and Amazon EMR, with shorter runtimes through elastic scaling
- Replaced Oozie with Airflow for workflow orchestration
- Designed and implemented several GitLab CI/CD pipelines
- Trained internal staff and transferred knowledge
Cloud-based data warehouse for user data analytics
Data Architect & Lead Engineer · Media · Jun 2020 – Dec 2022
- Designed and built the cloud infrastructure for the data warehouse
- Introduced a decentralized data mesh architecture, including governance guidelines
- Documented architecture decisions as ADRs and presented them to the architecture board
- Integrated Airflow as the workflow engine and the Spark on Kubernetes operator as the ETL runtime
- Built and led a 6-person team developing ETL processes for 4 data sources; implemented several ETL jobs in PySpark
- Implemented an event streaming pipeline for real-time analytics, including a real-time Braze integration with under 1 second latency
- Prepared data for downstream analysis in Power BI and advised on the overall data architecture
Sensor data processing platform and extension of a data science environment
- Advised on the design and tooling of a Kubernetes-based infrastructure for sensor data processing
- Designed and built the infrastructure on Kubernetes (Kafka cluster, Spark, ZooKeeper)
- Designed and built the ETL pipeline for data ingestion
- Designed and built a CI/CD pipeline with Bamboo and Kubernetes
Cloud-based data warehouse for vehicle data; evaluation of a data science environment
- Extended a prototype for large-scale data processing and brought it into production, improving runtime by a factor of 10
- Built and launched a CI/CD pipeline; set up project structure and release management
- Designed, implemented and deployed a backend API, including a Helm chart
- Built ETL pipelines for vehicle data validation, ingestion and indexing of millions of documents in OpenSearch
- Implemented feature extraction from vehicle data
- Evaluated Kubeflow and Databricks/Delta Lake
Cloud-based data warehouse and data science environment
- Designed a dynamically scalable data warehouse and its data model and storage layout
- Implemented the infrastructure as code with CloudFormation and infrastructure components on Kubernetes
- Built ETL pipelines for data ingestion
Design and implementation of a microservice architecture
- Aligned and coordinated several teams on technology choices
- Co-designed core elements: (de)serializers, data pipeline design, error and message handling, database design
- Introduced Kafka as the central message bus for microservices
- Introduced Liquibase for database schema management
- Provided functional and technical ownership of one microservice
Evaluation of big data vendors
- Gathered and documented technical and business requirements for building and operating a Hadoop-based data warehouse
- Requested vendor proposals and prepared the information for decision-making
- Implemented a data ingestion prototype
Big data warehouse on AWS
- Led the project technically and designed the AWS cloud infrastructure
- Implemented data pipelines, the data warehouse and workflow management
- Handled data preparation and process management
Book a free 30-minute call or email hello@crichter.io.
Experience
Ingenieurbüro Christian Richter
Freelance Data & AI Engineer · Jan 2010 – Present
- Independent consultancy for cloud data platforms, ETL and streaming, Kubernetes and infrastructure as code
- 25+ client engagements; see selected projects
Shibui Finance
Founder · Sep 2025 – Present
- Built and operate a public MCP server (mcp.shibui.finance) that makes 60+ years of US stock market data queryable in plain English from Claude; listed in the official MCP Registry
- Own data platform (dbt, PostgreSQL, DuckDB) with governed LLM access (read-only SQL, EXPLAIN validation, rate limiting)
- AI-driven software factory with Claude Code agents (plan → implement → review)
AltusInsight GmbH
Founder & Managing Director · Mar 2013 – Jun 2021
- Founded and ran the company for eight years; responsible for product design and implementation
- Built LambdaNow, a cloud-based platform for fully automated deployment of Apache Hadoop clusters on AWS and OpenStack
MOG Inc.
Backend Software Engineer & Systems Architect (contract) · May 2005 – Oct 2009
- First engineer at this music and media startup; responsible for backend software, databases and infrastructure
- Designed, built and launched the production hosting environment
- Developed C++ backend systems, including a distributed collaborative filtering engine, a search spellchecker and a high-performance XML-RPC server
Fraunhofer IDMT
Research Assistant · Aug 2000 – Mar 2005
- Researched nearest-neighbor and similarity search algorithms for music information retrieval (query by humming)
- Designed and implemented algorithms in C++ and MATLAB; built test environments and trade-show demo applications
ID Analytics Inc.
Data Analyst Intern · Jan 2003 – Jun 2003
- Analyzed ~250 GB of data for feature extraction and developed regression-based algorithms to detect identity theft
Skills
Core Stack
Python, SQL, Spark, Kafka, Airflow, dbt, Terraform, Kubernetes, AWS, GCP, MCP
AI & LLM
Vertex AI, Document AI, Model Context Protocol (MCP), llama.cpp, embeddings and vector search, AI-driven workflows, Claude Code, GitHub Copilot
Data Engineering
Spark/PySpark, Databricks, Delta Lake, Hudi, dbt, Airflow, AWS Glue, Apache NiFi, Pandas, Polars, NumPy, scikit-learn, Power BI
Streaming
Kafka, Kafka Connect, Kafka Streams, Strimzi, Amazon Kinesis, Apache Pekko
Databases & Formats
PostgreSQL, OpenSearch, MongoDB, DynamoDB, DuckDB, TimescaleDB, MySQL/MariaDB, Oracle, Hive, Impala, HBase, Cassandra; Parquet, Avro, ORC
Cloud
AWS (S3, EMR, Glue, Athena, Lake Formation, Lambda, Kinesis, RDS, DynamoDB, ECS/ECR, IAM, CloudFormation, CDK), Google Cloud (GKE, Vertex AI, Document AI), OpenStack
Containers
Kubernetes, OpenShift, Docker, Helm, Kustomize, Argo CD, cert-manager, Prometheus, Grafana, Kubeflow
IaC & DevOps
Terraform, Terragrunt, CDKTF, Ansible, GitOps, GitLab CI, GitHub, Jenkins, Bamboo, Open Policy Agent, Puppet, Chef
Programming
Python, Java, Kotlin, SQL, Scala, Bash, Ruby, C++; Spring Boot, Flask, Liquibase, Flyway, Maven, Gradle
Education
Technische Universität Ilmenau, Germany
Diplom-Ingenieur (Dipl.-Ing.), Electrical Engineering, equivalent to M.Sc. · Oct 1998 – Sep 2004
Major: Theoretical Electrical Engineering
Earlier projects (2010 – 2016)
Workshop on big data technologies
- Delivered a three-day workshop introducing the big data/Hadoop ecosystem, with hands-on exercises on AWS
Architecture review and real-time streaming layer
- Reviewed the existing architecture and data model; ran a workshop on data management and Lambda architecture
- Designed and implemented a real-time layer with Spark Streaming and integrated it with the batch layer
ETL pipelines for a Hadoop-based data warehouse
- Built a Hive export module and Kafka and Redis export modules for the open-source project Schedoscope
- Implemented an algorithm for clickstream analysis
Continuous delivery pipeline for a data-driven application on AWS
- Planned and built a big data infrastructure on AWS and a continuous deployment pipeline
- Led the client's internal team technically
Big data warehouse for OLAP workloads
- Planned and built the cluster infrastructure, benchmarked input formats for performance, and prepared and ran load tests
Batch and real-time processing of machine data
- Implemented a distributed, fault-tolerant, high-throughput messaging and analytics system for machine data (Lambda architecture)
- Planned the deployment environment, evaluated data ingestion and processing technologies, and led the team technically
Hadoop-based data warehouse for game analytics
- Planned and built the data warehouse, including a distributed, fault-tolerant, high-throughput messaging system
- Led and coordinated a distributed team across Germany, China and Canada
Big data infrastructure in virtualized environments
- Built a deployment process for on-demand Hadoop clusters on OpenStack
- Prototyped several algorithms in MapReduce
Big data architecture for telecommunications data
- Planned and built the network (AWS VPC) and a 100 TB Hadoop cluster, including deployment and monitoring
- Implemented a data ingestion framework storing ~300 GB of data per day
Design and rollout of a Hadoop cluster
- Advised on the cluster design, selected the hardware, set up the deployment process and rolled out the cluster
- Ported existing statistics routines to MapReduce
Re-implementation of an analytics tool as a MapReduce application
- Ported an existing implementation to MapReduce using the Hadoop Streaming API
- Installed and configured a Hadoop cluster with monitoring and set up the deployment process
Payment provider integration
- Integrated a payment provider into the existing backend
- Mapped, converted and imported data, including text matching against the existing database
Signature component for an email client
- Integrated a digital signature component into KMail, including debug environment setup and testing
Additional information
Patent
Co-inventor of European patent EP1377924 B1 (Fraunhofer IDMT)
Open Source
caddy-jsonrpc-matcher: Caddy module for selective rate limiting of MCP tool calls; Kafka and Redis export modules for Schedoscope, an open-source Hadoop data warehouse framework
Languages
German (native), English (fluent, full professional proficiency)
Work model
Remote or on-site, hybrid preferred (based in Berlin)
Profiles
Services
AI integration
AI-assisted data processing: document extraction, semantic search and workflow automation with LLM tool integration (MCP), including a self-built MCP server in production. See agent-ready data.
Data platforms
Architecture and delivery of data warehouses, data lakes and lakehouse platforms (AWS, GCP, Kubernetes), from concept to production
Data pipelines & streaming
Design and operation of ETL/ELT, streaming and ML pipelines (Spark, Kafka, Airflow, dbt) at terabyte scale
Cloud infrastructure as code
Cloud-native infrastructure as code (Terraform, Helm, GitOps), including high-availability and disaster-recovery concepts
Data governance & modeling
GDPR-compliant data management including data residency, data modeling, and requirements and process analysis
Technical leadership
Technical leadership of distributed teams, architecture reviews and knowledge transfer
Workshops
Hands-on workshops for your team on cloud data platforms, data pipelines, streaming and AI integration, tailored to your team's level. See workshops for topics.
More on the services page: common data pipeline patterns, how an engagement works, and workshops.
The first conversation is free: we talk through your needs and possible solutions, with no obligation to book me.
Book a free 30-minute callor email hello@crichter.io
Social links
- Github: https://github.com/christianrichter
- LinkedIn: https://www.linkedin.com/in/christian-richter-berlin/