Mahmoud Elshenhab

Staff Engineer | Platform, Reliability & Security Automation

View ATS-friendly resume العربية

Professional Summary

Staff engineer with 15 years across platform engineering, SRE and security automation at AWS, ServiceNow and Okta, with deep roots in Linux operations and relational databases. At Okta, designed and led Heimdall, a continuous compliance-evidence platform covering 171 security controls across Okta, GitHub and AWS; in one SOC 2 audit cycle it replaced ~635 engineering hours of manual work with 8 hours of automation, with 98% of evidence accepted by auditors. Before that, three years as a Staff SRE on call for ServiceNow's entire customer instance fleet, and six at AWS - first as an RDS subject-matter expert, then on infrastructure-as-code and region-build automation, including the project that cut fleet patching from 40+ days to 7. Works AI-first, building agent tooling and multi-agent systems that scale engineering output.

Professional Experience

September 2024 - Present

Staff Compliance Automation Engineer

Okta | Dublin, Ireland (Remote)

  • Designed and led Heimdall, Okta's continuous compliance-evidence platform, as tech lead and primary author - ~90% of 570+ commits, ~33k lines of production Python and 2,100+ automated tests.
  • Automated a SOC 2 audit cycle end to end - 127 evidence tickets collected, attached and submitted with no manual work, 125 accepted by auditors (98%) and the other 2 routed to service owners for remediation, replacing ~635 engineering hours of manual collection (~5 hours per ticket) with 8 hours of automated runtime.
  • Architected a plugin-based core (orchestration, credential management, per-call API audit trail, multi-format reporting) that new data sources plug into without touching core logic, and scaled it to 23 collectors across Okta, GitHub, AWS and HTTP producing 129 evidence artifacts mapped to 171 security controls.
  • Closed the loop with the audit process - built a Windmill-based orchestrator that claims audit Jira tickets, runs the matching collection, attaches the evidence and transitions the ticket, with stale-job detection, Slack alerting with one-click retry and staggered monthly schedules per artifact.
  • Built an integrity-verified evidence delivery pipeline - 5 AWS Lambda functions orchestrated by Step Functions and defined in CloudFormation, verifying checksums at every hop from S3 to Google Drive.
  • Owned CI/CD and release architecture - GitHub Actions builds one version-locked artifact, verified at three independent points, auto-deployed to dev and promoted weekly to production via AWS CodeDeploy (65 tagged releases), with zero-touch host provisioning and self-bootstrapping credentials.
  • Delivered the Heimdall and Windmill platforms as infrastructure as code - AWS resources defined in CloudFormation, Windmill deployed as Docker containers, and everything rolled out to hosts with AWS CodeDeploy and driven end to end by GitHub Actions, so both environments are built from the repository rather than by hand.
  • Implemented secure-by-design API clients (OAuth2 private-key JWT with DPoP, GitHub App authentication, rate-limit-aware pagination) and closed security and reliability gaps - remediated Dependabot findings, hardened serialization paths and re-architected reporting to stream to disk, eliminating out-of-memory failures on large collection runs.
  • Built ActionHub, a compliance workflow platform (Flask / PostgreSQL) that replaced manual compliance questionnaires - dynamic forms, assignment lifecycle, reviewer RBAC, a background workflow engine, full audit logging and Okta SSO, backed by 980 tests.
  • Drove AI adoption across the team - authored Claude Code skills that make evidence tracing and ticket triage deterministic, built local MCP servers (generated with Claude Code) for internal APIs such as Jira so agents could operate those services directly, ran sessions on AI-assisted engineering, and designed a multi-agent system that generates new collectors from Jira requests to scale coverage across the remaining audit scope.
  • Mentored engineers on secure, testable software design, documented the platform's architecture and data flows in the repository with Mermaid diagrams, and took it to steady state before handing long-term ownership to Security Architecture.
September 2021 - August 2024

Staff Site Reliability Engineer

ServiceNow | Dublin, Ireland (Remote)

  • On-call Staff SRE for ServiceNow's entire fleet of customer production instances - a continuous, high-volume incident load with no quiet shifts - handling incident response, alert triage and escalation alongside customer support and engineering teams, using Splunk and Grafana for investigation and monitoring.
  • Automated recurring alert handling, reducing manual triage and improving response accuracy.
  • Improved platform stability and performance through internal automation services, and tightened the team's runbooks and operational procedures.
December 2019 - September 2021

System Development Engineer

Amazon Web Services (AWS) | Dublin, Ireland

  • Led the patching-automation project across several teams' fleets, cutting patching cycle time from 40+ days to 7, and served as the subject-matter expert for patching workflows.
  • Started zero-touch build workflows for bastion fleets, removing manual host configuration.
  • Diagnosed and fixed a critical fleet-monitoring defect involving the service's Gossip protocol.
  • Automated region builds and service deployments for an AWS networking service as infrastructure as code, defining the service's pipelines and infrastructure in an internal Ruby-based templating tool, improving deployment speed and consistency.
  • Took over and ran a production fleet's Puppet-based configuration management - operated the pipeline that pushed manifests to the hosts, and Puppet on each host applied them every 4 hours, so the fleet continuously converged to the declared state and drift was corrected automatically.
  • Owned production fleet reliability on the on-call rotation - incident response, maintenance and feature development.
May 2018 - November 2019

System Engineer I

Amazon Web Services (AWS) | Dublin, Ireland

  • Built region-build and deployment automation for an AWS networking service in Ruby using an internal infrastructure-as-code templating tool, improving consistency and reducing release cycle times.
  • Joined the on-call rotation for production fleet reliability, maintenance and incident response; promoted to System Development Engineer.
October 2017 - April 2018

Cloud Support Engineer II

Amazon Web Services (AWS) | Cape Town, South Africa

  • Resolved complex infrastructure and database issues for AWS Enterprise customers, specializing in RDS, and led root-cause analyses for customer and internal service teams.
  • Ran MySQL, MariaDB and Aurora migrations across AWS regions with minimal downtime, and advised on highly available database architectures with AWS Solutions Architects.
November 2015 - September 2017

Cloud Support Engineer I

Amazon Web Services (AWS) | Cape Town, South Africa

  • Recognized as an RDS Subject Matter Expert; ran cross-region MySQL, MariaDB and Aurora migrations with under 5 minutes of downtime.
  • Discovered bugs in internal AWS workflows that affected thousands of customers, and wrote scripts to reproduce them consistently for the service team, resulting in a patch to the entire RDS fleet.
  • Designed hands-on onboarding training for MySQL, MariaDB and Aurora support, and mentored 6 engineers to subject-matter-expert level.
  • Advised customers on database architecture and cost - steering product choices to their workload and budget, and persuading several internal and external customers to redesign how their applications communicated with the database, significantly improving workload performance.
October 2014 - September 2015

System Engineer

Freelance | Cairo, Egypt

  • Took on system administration and engineering contracts, including re-architecting a local company's Access-based management system to work across multiple branches.
  • Used the period to complete the University of Illinois Cloud Computing Concepts courses (CCC1, CCC2) ahead of joining AWS.
March 2012 - October 2014

NIX System Engineer

Link Data Center (LinkDev) | Cairo, Egypt

  • Built and ran the CentOS/RHEL hosting platform behind msnarabia.com, serving 7M+ hits a day, and led the company-wide migration from Apache HTTP Server to Nginx.
  • Designed distributed media hosting with Lsyncd in place of NAS/SAN, removing single points of failure at lower cost, and set up active-active MySQL binary-log replication for database high availability.
  • Automated the migration of customer DNS zones from Windows DNS to BIND and the monitoring of the replication service, both in Bash.
October 2011 - March 2012

System Administrator

International Turnkey Systems (ITS) | Cairo, Egypt

  • Administered Windows Server and Linux on VMware ESX, plus Microsoft SQL Server and Sybase ASE databases.
  • Built a centralized, automated backup and rotation solution for all database servers in Bash and PowerShell.

Education

September 2005 - June 2009

Bachelor of Computer Sciences

Misr Higher Institute for Commerce & Computers | Mansoura, Egypt

Grade - Good (70.73%)

Certifications

Microsoft Certified - Azure Fundamentals (AZ-900)

Issuer: Microsoft • Date: October 22, 2021 • ID: I003-7698

View certificate

Introduction to Computer Science and Programming Using Python

Issuer: MITx (edX) • Date: March 17, 2017 • ID: 05a576b4853640b0bfe7b9b221a9fd5d

View certificate

Cloud Computing Concepts, Part 2 (CCC2)

Issuer: University of Illinois at Urbana-Champaign (Coursera) • Date: April 27, 2015 • ID: 48W63NBRVF

View certificate

Cloud Computing Concepts, Part 1 (CCC1)

Issuer: University of Illinois at Urbana-Champaign (Coursera) • Date: March 17, 2015 • ID: 4LVSBUDRWC

View certificate

Red Hat Certified System Administrator (RHCSA) - RHEL 5 (EX200)

Issuer: Red Hat • Date: April 16, 2009 • ID: 100-108-729

View certificate

Languages

Arabic

Native

English

Fluent (bilingual professional proficiency)

Projects

Football Wizards - Football Analytics Platform (live at api.gurutux.com)

Designed and operate a two-service FastAPI / SQLAlchemy / PostgreSQL platform - a public read-only API with a React explorer, and an isolated admin control plane for scheduled ingestion jobs with live progress streaming. Modelled a normalized schema (123 player and 32 team stat types, minute-stamped match events) holding 10M+ stat rows across ~9,600 matches, fed by a pluggable ingestion layer with 5 source adapters, content-hash caching and cross-source entity resolution. Enforced least privilege at the database layer with separate Postgres roles, so the public service cannot read admin state or secrets. Built a validation framework that caught a column-mapping defect and corrected 5,307 records. Shipped with Docker Compose, auto-applied Alembic migrations and one-command deploys with health checks, self-hosted on Proxmox. Developed AI-first with Claude Code under a persisted architecture and decision log.

Automated Azan - Prayer Announcements for Chromecast (github.com/shenhab/Automated-Azan)

Open-source Python service that schedules and streams prayer announcements to Chromecast and Google Home devices, with a web management interface. Packaged as a multi-stage Docker image (non-root user, health check) published to Docker Hub, with a GitHub Actions release pipeline that builds and pushes versioned and latest tags on each release and a Docker Compose deployment.

YamlCV

This CV. A Jekyll-based CV-as-code system that uses YAML as the single source of truth and generates both an interactive HTML resume and an ATS-friendly version from it. https://github.com/shenhab/YamlCV

Home Lab - Self-Hosted Infrastructure

Run a self-hosted environment on a two-node Proxmox VE cluster that lets workloads be managed and migrated between nodes, with a TrueNAS server providing shared storage and continuous off-site backup of its data to cloud storage. It hosts my own services, including Football Wizards and an n8n instance for workflow automation.

Skills

Languages

  • Python
  • Bash
  • SQL
  • Ruby

Platforms & Delivery

  • GitHub Actions
  • AWS CodeDeploy
  • Windmill
  • n8n
  • Slack integrations

Infrastructure as Code

  • CloudFormation
  • Puppet
  • Continuous configuration convergence
  • Zero-touch host provisioning

Containers & Virtualization

  • Docker
  • Docker Compose
  • Container image builds
  • Proxmox VE (clustered)
  • TrueNAS
  • Self-hosted infrastructure
  • Kubernetes (fundamentals)

AWS

  • Lambda
  • Step Functions
  • EC2
  • S3
  • RDS / Aurora

Reliability & Operations

  • On-call & incident response
  • Alert automation
  • Linux (RHEL / CentOS)
  • Nginx
  • DNS
  • Grafana
  • Splunk
  • SLIs / SLOs / SLAs & error budgets (fundamentals)

Distributed Systems

  • Gossip protocols & failure detection
  • Replication & consistency models
  • Fault tolerance & incident analysis
  • Consensus & leader election (CCC1 / CCC2)
  • MapReduce & stream processing (CCC1 / CCC2)

Backend & Data

  • FastAPI
  • Flask
  • SQLAlchemy / SQLModel
  • Alembic
  • PostgreSQL
  • MySQL
  • MariaDB

Security & Identity

  • Okta
  • OAuth2
  • Private-key JWT / DPoP
  • GitHub App auth
  • RBAC
  • Least-privilege design
  • Audit trails

Compliance Automation

  • Control-to-evidence mapping
  • Continuous evidence collection
  • Audit workflow automation
  • Evidence integrity verification

AI & Agentic Engineering

  • Claude Code
  • Agent skills & hooks
  • MCP
  • Multi-agent system design

Docs as Code

  • Mermaid
  • YamlCV