Mohammed Yousuf Lohia

Project — Operational Data Hub

• Designed and implemented an Operational Data Hub on Azure using Medallion Architecture, integrating multiple source systems.
• Designed and implemented v2 of the Data Hub in AWS with support for near real-time data ingestion and dissemination. V2 leveraged AWS MSK and Apache Hudi to support streaming ingestion and end-to-end streaming using the data lake.
• Leveraged Azure Data Factory, Databricks, and Synapse Analytics to build an end-to-end data pipeline for orchestration, transformation, and reporting — using ADF to orchestrate movement from multiple source systems, transforming through Bronze → Silver → Gold layers in Databricks, and delivering curated datasets to Synapse for enterprise reporting.
• Designed each pipeline to be metadata-driven and configurable to run at intervals of 1, 3, or 12 hours based on data characteristics.
• Worked with Data Architecture & Delivery teams to ensure the physical data model was captured in Erwin and representative of the target database, with table structures aligned to attribute naming conventions and architecture best practices. Implemented a UUID generation strategy re-creatable from an enum value to support parallel processing and re-loading into the target database.
• Developed CDC (Change Data Capture) patterns and built Delta Lake tables across Bronze, Silver, and Gold layers in Databricks. Created consumption-ready views in Synapse Analytics for Power BI reporting and implemented Dynamic Data Masking to protect sensitive attributes based on RBAC role assignments.
• In AWS, designed an architecture using Debezium connectors on MSK Connect to stream CDC events onto an ingestion topic and consume the data into the data lake for long-term retention, mirroring ingestion topics to consumer topics for near real-time dissemination — implemented in six months.
• Used Databricks for data engineering workloads and cluster management, and AWS Lake Formation for tag-based and role-based access controls enabling secure data sharing.
• Set up an Azure DevOps project with managed identities, service principals, and RBAC roles to securely manage read, write, and admin access to code repositories. Implemented a multi-repo strategy separating infrastructure-as-code from data pipeline code for modular development and cleaner CI/CD.
• Defined and enforced a branching strategy (main, dev, feature/Sprint-*) to maintain a consistently stable dev branch while supporting parallel feature development.
• Implemented multi-stage YAML pipelines with ARM templates in Azure DevOps to automate environment promotion with integrated approval gates for production, including an orchestrator pipeline to tear down and recreate entire environments for mock deployments and UAT.
• Centralized Synapse and Delta Lake table definitions in Git, enforcing deployment exclusively via the workload deployment pipeline for consistency and environment parity.
• Designed a workload deployment pipeline in Azure DevOps to orchestrate releases across Synapse, Databricks, and Data Factory using SSDT, shell scripts, and PowerShell, with a standardized CD template for repeatable deployments.
• Designed an auto-shutdown pipeline to turn off non-critical Azure resources outside business hours, resulting in annual cloud cost savings of $100K–$150K.
• Developed custom log- and metric-based Azure Alerts to proactively detect anomalies and workload failures, monitored global service health, and automated ITSM incident creation for faster response and root-cause analysis, with post-recovery reviews to reduce recurrence.
• Configured Azure Alerts for near-expiry and expired secrets to enable proactive rotation and reduce risk of service disruption.
• Developed Azure DevOps pipelines to replicate data from production to non-production environments, enabling realistic testing and faster issue resolution.
• Implemented Azure Log Analytics Workspace and Application Insights, streaming logs for centralized logging, advanced querying, and dashboard creation.
• Developed Data APIs for internal systems to establish a trusted, centralized data source for data-driven applications.
• Demonstrated approximately $300K in annual infrastructure savings by right-sizing Databricks clusters and validating no performance impact.
• Completed designs for integration with Confluent Kafka to support near real-time data dissemination.

Project — Enterprise IT Planning / Delivery Support

• Participated in SOW reviews, providing feedback on scope, delivery assumptions, and out-of-scope deliverables.
• Presented solution designs to architecture stakeholders and the review board for alignment with existing standards, and documented new standards to enable future projects. Participated in design reviews for other projects requiring integration with the Operational Data Hub.
• Participated in weekly stakeholder meetings representing technical accountability for in-flight projects, articulating risks, actions, issues, and dependencies.
• Worked closely with program managers to provide development estimates using top-down and bottom-up approaches.

Project — Data Assets Landscape Re-Architecture

• Documented the current IT landscape used by the Data Assets group to maintain and support 100+ ETL jobs that ingest, transform, and produce data products used by internal and external consumers.
• Established a framework to catalog jobs by type and other metadata, providing a complete picture of the IT landscape, external dependencies, third-party tools, licenses, and the nature of ETL jobs in the environment.
• Established a target-state architecture to support both data and spatial analytics capabilities identified during the review, and presented it to stakeholders as part of ITDM.