Success Story
PETRONAS AWSM — Downstream Petroleum Scheduling
Real-time dashboards, ETL workflows, SAP integration, and WebSocket APIs for downstream petroleum distribution at a Fortune 500 oil and gas operator.
Challenge: Downstream petroleum distribution scheduling at PETRONAS depended on batch reports and disconnected SAP exports. Operations teams could not see shipment status in real time, and the data science team needed a clean pipeline to plug in forecasting models without fighting infrastructure.
Solution: Led the backend and DevOps team building AWSM, a downstream scheduling platform with Lambda + API Gateway services feeding real-time dashboards, Step Functions orchestrating ETL across multiple data sources into MySQL and the data warehouse, and WebSocket APIs streaming SAP shipment events to the operations console.
Result: Operations team got real-time shipment visibility on one console. ETL workflows ran on their own. The data science team plugged in their forecasting models against a clean data layer. EC2 spend stayed flat as scope grew.
Tech Stack
The Story
PETRONAS ICT brought me in as Backend and DevOps Lead for AWSM, the downstream petroleum distribution scheduling platform. The team was strong on the operations side but needed architecture direction on AWS and Terraform. I set the technical direction, ran code reviews, and mentored engineers on serverless patterns and infrastructure as code.
The first chunk of work was getting real-time visibility into shipment status. The existing flow pulled SAP data on a schedule and dumped it into reports. I built WebSocket APIs that stream shipment events as they happen, plus Lambda functions and API Gateway endpoints feeding the operations dashboard. When a truck loads or a shipment dispatches, the console updates immediately. No more refreshing a stale report.
The second chunk was the ETL layer. Data was landing in MySQL and the warehouse from multiple sources, each with its own format and cadence. I wrote Step Functions workflows that handle the orchestration: extract, transform, load, with retries and error handling so a bad source row does not take down the whole pipeline. Once the workflows were stable, the data science team could plug their forecasting models in against a predictable schema instead of fighting input quality.
The third chunk was cost. EC2 was the bulk of the spend, and the data science workloads were hungry. I worked with the Data Science team to right-size the instance types they were using and shift the batchable jobs onto cheaper compute. The spend stayed flat as we added more dashboards and more model runs.
I also ran the stakeholder side. Presented roadmaps and progress to leadership, kept everyone aligned on what was shipping and when. Twelve-month engagement, downstream operations team that actually used what we built, and a data layer the data science team could build on instead of work around.
How We Delivered
Our Delivery Process
See how our senior engineering pod delivered production-ready results
Real-time Operations Dashboard
- WebSocket APIs streaming shipment events from SAP to the operations console as they happen, replacing scheduled batch reports.
- Lambda plus API Gateway services backing the dashboard with sub-second response times on shipment status queries.
- SAP integration handling the format and cadence quirks of the existing operational systems so the dashboard sees clean events.
Step Functions ETL
- Orchestrated ETL pipelines pulling data from multiple sources into MySQL and the data warehouse with retries and per-step error isolation.
- Predictable schema downstream so the data science team could plug forecasting models in without fighting input quality.
- Idempotent step design preventing duplicate loads on retries during pipeline failures.
Cost & Team Leadership
- EC2 cost optimization with the Data Science team, right-sizing instance types and shifting batchable jobs onto cheaper compute. Spend held flat as scope grew.
- Set architecture direction on AWS and Terraform for the backend and DevOps team. Ran code reviews and mentored engineers on serverless patterns.
- Presented roadmaps and progress to stakeholders, kept operations and data science teams aligned on delivery.
Final Outcomes
Results
Working on something similar?
Book a 15-minute call. We'll tell you honestly if we're the right fit.
Book a 15-min Call