Hi folks, I’ve written a Prometheus exporter for the BikePoint API to track availability over time. The idea was to potentially raise notifications when a given station is below a threshold of available bikes or docks.
My hypothesis is this happens when a subset of backend processes stop receiving new data, causing them to serve the same values over and over again. From this point on, a given API call randomly receives either current or stale numbers, causing the oscillation when graphed over time. This would explain why the “wrong” value is a constant equal to the correct value at the time the pathological behaviour starts.
I’m seeing the same results when querying the API from two different data centres in London. The behaviour also correlates with slightly lower p99 response times, I suspect brought down by the bad values. Perhaps the backend is reading from a fast but stale cache rather than retrieving from the source:
Thanks, in case it helps, these are the periods where the behaviour has happened over the past 12 weeks, using the number of changes in bikes available for one station as a heuristic:
Hi @gebn - we found some intermittent errors in our BikePoint ETL task. We believe the caching layer in front of the source file is causing issues. We’ve bypassed this layer in our dev environment and it seems to be a lot more reliable.
We’ll look to get this in production next week and I’ll give you a further update then.
