tRS Notifications

Incident Update Page

Issue: Portal Access

Current Status = Back online

Date: Tuesday 5, November 2024 · All times in AEST

Activity Log

  1. 09:48Alert – An outage was logged accessing the production portal for store and head office users
  2. 09:55Escalation – Infrastructure and Application Development Team
  3. 10:03All Hands – Co-ordinated meeting to discuss actions and communications
  4. 10:06Comms – Universal Incident Page Created
  5. 10:10Testing – Fix being applied and tested
  6. 10:16Fix Applied – applying fix to production platform
  7. 10:20Comms – Informing clients portal is coming back on
  8. 10:25Root Cause – Commence root cause analysis
  9. 10:30Fix Confirmed – Working on root cause analysis
  10. 10:45Root Cause Analysis Complete (See Below)

Next Update 10:45 – Incident closed. No further updates

Root Cause Analysis

We have completed our root cause analysis into the portal outage in this incident report.

The Cause

Every time the portal is queried, a log entry is generated to allow TRS to audit cube usge to help with identifying any long running queries or issues. The logs are loaded into the usage database every 5 minutes and this has been running successfully for over 6 years however this process stopped running causing the log queue to reach capacity at approximately 926am AEST.

The Immediate Fix

Once we restarted the queue reading process there was enough space for additional logs to be loaded and the portal was restored.

The Permanent Fix

We will be making two adjustments to this process. 1. The logging process will be downgraded to a non-critical so this will no longer impact the availability of the production site if it fails. 2. A new alert has been written to pre-emptively inform the monitoring team of the capacity of the portal-log database.

On the positive, the new central incident page was activated and although this process will improve in time, it did allow us the facility to centrally publish updates in near real-time.