Resolved - A drop in IP volume was triggered by an internal release midday on August 15th 2019. During said release an erroneous configuration line was included. The erroneous configuration line was responsible for data being written to a QA environment instead of a production environment. Existing monitoring and notifications were in place to alert but not configured properly to capture this specific case. As a result of data being diverted to QA, IP addressed were allowed to expire from our production system as per design. With each incremental publishing of data an incremental drop in volume was experienced until the issue was resolved. Note: IPs are typically added to the dataset for a short duration (roughly 1 to 5 days on average) depending on severity and impact. The largest volume of drop in IP counts occurred only after addresses held for >= 3 days were allowed to expire from the data set.

Response timeline in PDT (Monday 8/19):
• 4:00-4:30 PM BI and Stats reporting alerted team to an IP variance
• 4:45 PM DevOps alerted to massive drop in IP volume
• 5:07 PM Cause of issue identified and corrected
• 5:20 PM Product notified of issue
• 6:01 PM Internal stakeholders and escalation process notified
• 7:00 PM Full build published (min rev 2155)
• 7:04 PM Notification new full build was published
• 1 hour For monitoring and change validation
Aug 19, 16:45 UTC

Resolved - This issue has been resolved.
Aug 12, 09:40 UTC