Skip to main content

Success Story

PETRONAS AWSM — Downstream Petroleum Scheduling

Real-time dashboards, ETL workflows, SAP integration, and WebSocket APIs for downstream petroleum distribution at a Fortune 500 oil and gas operator.

Real-timeShipment visibility
12-month engagement, PETRONAS ICT
AWS LambdaAPI GatewayStep FunctionsWebSocket APIsSAP IntegrationMySQLData WarehouseEC2TerraformCloudWatchNode.jsPython
12 months
Engagement
Real-time WebSocket
Shipment visibility
Step Functions
ETL orchestration
Fortune 500 Oil & Gas
Sector

Challenge: Downstream petroleum distribution scheduling at PETRONAS depended on batch reports and disconnected SAP exports. Operations teams could not see shipment status in real time, and the data science team needed a clean pipeline to plug in forecasting models without fighting infrastructure.

Solution: Led the backend and DevOps team building AWSM, a downstream scheduling platform with Lambda + API Gateway services feeding real-time dashboards, Step Functions orchestrating ETL across multiple data sources into MySQL and the data warehouse, and WebSocket APIs streaming SAP shipment events to the operations console.

Result: Operations team got real-time shipment visibility on one console. ETL workflows ran on their own. The data science team plugged in their forecasting models against a clean data layer. EC2 spend stayed flat as scope grew.

Tech Stack

AWS LambdaAPI GatewayStep FunctionsWebSocket APIsSAP IntegrationMySQLData WarehouseEC2TerraformCloudWatchNode.jsPython

The Story

PETRONAS ICT brought me in as Backend and DevOps Lead for AWSM, the downstream petroleum distribution scheduling platform. The team was strong on the operations side but needed architecture direction on AWS and Terraform. I set the technical direction, ran code reviews, and mentored engineers on serverless patterns and infrastructure as code.

The first chunk of work was getting real-time visibility into shipment status. The existing flow pulled SAP data on a schedule and dumped it into reports. I built WebSocket APIs that stream shipment events as they happen, plus Lambda functions and API Gateway endpoints feeding the operations dashboard. When a truck loads or a shipment dispatches, the console updates immediately. No more refreshing a stale report.

The second chunk was the ETL layer. Data was landing in MySQL and the warehouse from multiple sources, each with its own format and cadence. I wrote Step Functions workflows that handle the orchestration: extract, transform, load, with retries and error handling so a bad source row does not take down the whole pipeline. Once the workflows were stable, the data science team could plug their forecasting models in against a predictable schema instead of fighting input quality.

The third chunk was cost. EC2 was the bulk of the spend, and the data science workloads were hungry. I worked with the Data Science team to right-size the instance types they were using and shift the batchable jobs onto cheaper compute. The spend stayed flat as we added more dashboards and more model runs.

I also ran the stakeholder side. Presented roadmaps and progress to leadership, kept everyone aligned on what was shipping and when. Twelve-month engagement, downstream operations team that actually used what we built, and a data layer the data science team could build on instead of work around.

How We Delivered

Our Delivery Process

See how our senior engineering pod delivered production-ready results

1

Real-time Operations Dashboard

  • WebSocket APIs streaming shipment events from SAP to the operations console as they happen, replacing scheduled batch reports.
  • Lambda plus API Gateway services backing the dashboard with sub-second response times on shipment status queries.
  • SAP integration handling the format and cadence quirks of the existing operational systems so the dashboard sees clean events.
2

Step Functions ETL

  • Orchestrated ETL pipelines pulling data from multiple sources into MySQL and the data warehouse with retries and per-step error isolation.
  • Predictable schema downstream so the data science team could plug forecasting models in without fighting input quality.
  • Idempotent step design preventing duplicate loads on retries during pipeline failures.
3

Cost & Team Leadership

  • EC2 cost optimization with the Data Science team, right-sizing instance types and shifting batchable jobs onto cheaper compute. Spend held flat as scope grew.
  • Set architecture direction on AWS and Terraform for the backend and DevOps team. Ran code reviews and mentored engineers on serverless patterns.
  • Presented roadmaps and progress to stakeholders, kept operations and data science teams aligned on delivery.

Final Outcomes

Results

Real-time shipment visibility on one operations console, replacing batch SAP reports
Step Functions ETL pipelines running autonomously with retries and error isolation
Data science team plugged in forecasting models against a clean, predictable data layer
EC2 spend held flat as dashboards and model runs grew
Backend and DevOps team upskilled on serverless and infrastructure as code
Stakeholder alignment maintained across operations, engineering, and data science

Working on something similar?

Book a 15-minute call. We'll tell you honestly if we're the right fit.

Book a 15-min Call

Free AWS Architecture Roadmap
48-hour delivery. $12K value.