StopPoint Arrivals is out of date if I use my app key

dev_rob · January 1, 2021, 2:26pm

Certain stops are returned old results from 26 December, but only if I use the app_id and app_key. Without that it returns live results. I’ve tried in a couple of browsers to check it’s not a local cacheing issue, and it matches the data my app is getting.

Here’s the url, I’ll remove the actual keys:
https://api.tfl.gov.uk/StopPoint/940GZZLUKSL/Arrivals?mode=tube&app_id=(removed)&app_key=(removed)

This returns

[
  {
    "$type": "Tfl.Api.Presentation.Entities.Prediction, Tfl.Api.Presentation.Entities",
    "id": "1059139851",
    "operationType": 1,
    "vehicleId": "222",
    "naptanId": "940GZZLUKSL",
    "stationName": "Kensal Green Underground Station",
    "lineId": "bakerloo",
    "lineName": "Bakerloo",
    "platformName": "Northbound - Platform 2",
    "direction": "outbound",
    "bearing": "",
    "destinationNaptanId": "940GZZLUHAW",
    "destinationName": "Harrow & Wealdstone Underground Station",
    "timestamp": "2020-12-26T10:28:48.1846152Z",
    "timeToStation": 1840,
    "currentLocation": "Between Elephant & Castle and Lambeth North",
    "towards": "Harrow and Wealdstone",
    "expectedArrival": "2020-12-26T10:59:28Z",
    "timeToLive": "2020-12-26T10:59:28Z",
    "modeName": "tube",
    "timing": {
      "$type": "Tfl.Api.Presentation.Entities.PredictionTiming, Tfl.Api.Presentation.Entities",
      "countdownServerAdjustment": "00:00:00",
      "source": "0001-01-01T00:00:00",
      "insert": "0001-01-01T00:00:00",
      "read": "2020-12-26T10:29:46.668Z",
      "sent": "2020-12-26T10:28:48Z",
      "received": "0001-01-01T00:00:00"
    }
  },
  {
    "$type": "Tfl.Api.Presentation.Entities.Prediction, Tfl.Api.Presentation.Entities",
    "id": "-2106697535",
    "operationType": 1,
    "vehicleId": "224",
    "naptanId": "940GZZLUKSL",
    "stationName": "Kensal Green Underground Station",
    "lineId": "bakerloo",
    "lineName": "Bakerloo",
    "platformName": "Northbound - Platform 2",
    "direction": "outbound",
    "bearing": "",
    "destinationNaptanId": "940GZZLUHAW",
    "destinationName": "Harrow & Wealdstone Underground Station",
    "timestamp": "2020-12-26T10:28:48.1846152Z",
    "timeToStation": 1420,
    "currentLocation": "Between Charing Cross and Piccadilly Circus",
    "towards": "Harrow and Wealdstone",
    "expectedArrival": "2020-12-26T10:52:28Z",
    "timeToLive": "2020-12-26T10:52:28Z",
    "modeName": "tube",
    "timing": {
      "$type": "Tfl.Api.Presentation.Entities.PredictionTiming, Tfl.Api.Presentation.Entities",
      "countdownServerAdjustment": "00:00:00",
      "source": "0001-01-01T00:00:00",
      "insert": "0001-01-01T00:00:00",
      "read": "2020-12-26T10:29:46.668Z",
      "sent": "2020-12-26T10:28:48Z",
      "received": "0001-01-01T00:00:00"
    }
  },
  {
    "$type": "Tfl.Api.Presentation.Entities.Prediction, Tfl.Api.Presentation.Entities",
    "id": "657977210",
    "operationType": 1,
    "vehicleId": "225",
    "naptanId": "940GZZLUKSL",
    "stationName": "Kensal Green Underground Station",
    "lineId": "bakerloo",
    "lineName": "Bakerloo",
    "platformName": "Southbound - Platform 1",
    "direction": "inbound",
    "bearing": "",
    "destinationNaptanId": "940GZZLUEAC",
    "destinationName": "Elephant & Castle Underground Station",
    "timestamp": "2020-12-26T10:28:48.1846152Z",
    "timeToStation": 58,
    "currentLocation": "At Platform",
    "towards": "Elephant and Castle",
    "expectedArrival": "2020-12-26T10:29:46Z",
    "timeToLive": "2020-12-26T10:29:46Z",
    "modeName": "tube",
    "timing": {
      "$type": "Tfl.Api.Presentation.Entities.PredictionTiming, Tfl.Api.Presentation.Entities",
      "countdownServerAdjustment": "00:00:00",
      "source": "0001-01-01T00:00:00",
      "insert": "0001-01-01T00:00:00",
      "read": "2020-12-26T10:29:46.668Z",
      "sent": "2020-12-26T10:28:48Z",
      "received": "0001-01-01T00:00:00"
    }
  },
  {
    "$type": "Tfl.Api.Presentation.Entities.Prediction, Tfl.Api.Presentation.Entities",
    "id": "-1705592170",
    "operationType": 1,
    "vehicleId": "231",
    "naptanId": "940GZZLUKSL",
    "stationName": "Kensal Green Underground Station",
    "lineId": "bakerloo",
    "lineName": "Bakerloo",
    "platformName": "Southbound - Platform 1",
    "direction": "inbound",
    "bearing": "",
    "destinationNaptanId": "940GZZLUEAC",
    "destinationName": "Elephant & Castle Underground Station",
    "timestamp": "2020-12-26T10:28:48.1846152Z",
    "timeToStation": 700,
    "currentLocation": "At Wembley Central Platform 2",
    "towards": "Elephant and Castle",
    "expectedArrival": "2020-12-26T10:40:28Z",
    "timeToLive": "2020-12-26T10:40:28Z",
    "modeName": "tube",
    "timing": {
      "$type": "Tfl.Api.Presentation.Entities.PredictionTiming, Tfl.Api.Presentation.Entities",
      "countdownServerAdjustment": "00:00:00",
      "source": "0001-01-01T00:00:00",
      "insert": "0001-01-01T00:00:00",
      "read": "2020-12-26T10:29:46.668Z",
      "sent": "2020-12-26T10:28:48Z",
      "received": "0001-01-01T00:00:00"
    }
  },
  {
    "$type": "Tfl.Api.Presentation.Entities.Prediction, Tfl.Api.Presentation.Entities",
    "id": "1552401836",
    "operationType": 1,
    "vehicleId": "233",
    "naptanId": "940GZZLUKSL",
    "stationName": "Kensal Green Underground Station",
    "lineId": "bakerloo",
    "lineName": "Bakerloo",
    "platformName": "Northbound - Platform 2",
    "direction": "outbound",
    "bearing": "",
    "destinationNaptanId": "940GZZLUHAW",
    "destinationName": "Harrow & Wealdstone Underground Station",
    "timestamp": "2020-12-26T10:28:48.1846152Z",
    "timeToStation": 880,
    "currentLocation": "At Edgware Road Platform 1",
    "towards": "Harrow and Wealdstone",
    "expectedArrival": "2020-12-26T10:43:28Z",
    "timeToLive": "2020-12-26T10:43:28Z",
    "modeName": "tube",
    "timing": {
      "$type": "Tfl.Api.Presentation.Entities.PredictionTiming, Tfl.Api.Presentation.Entities",
      "countdownServerAdjustment": "00:00:00",
      "source": "0001-01-01T00:00:00",
      "insert": "0001-01-01T00:00:00",
      "read": "2020-12-26T10:29:46.668Z",
      "sent": "2020-12-26T10:28:48Z",
      "received": "0001-01-01T00:00:00"
    }
  }
]

If I remove the keys, then I get live results. (As I write there are no trains on the Bakerloo line so its empty, but this worked earlier)
https://api.tfl.gov.uk/StopPoint/940GZZLUKSL/Arrivals?mode=tube

Is this a known issue? Is there a way to solve it?

Rob

briantist · January 1, 2021, 5:01pm

If it’s caching problem then the thing to is to add an extra dummy parameter on the end, ie &dontcacheme=1234565

dev_rob · January 1, 2021, 8:42pm

That’s a good idea. Yep, just adding &x on the end seems to fix this. So there’s some sort of server-side cache happening which isn’t updating.

But, is this a workaround, or should the content be updating without this? The results used to update reliably, it just seems to have stuck on Boxing Day, and I’d rather not do a workaround.

briantist · January 2, 2021, 9:46am

Of course it … should be … operating without this.

The problem we have here is that there can be “invisible” caches in various systems, from the Browser environment to a “cloud” cache at the server farm end.

When you make the https://api.tfl.gov.uk/StopPoint/940GZZLUKSL/Arrivals?mode=tube request, the response headers look like this will all the cache values set to “no-cache”

so you would expect this to instruct the caches between your code - via the framework, the Browser, the CDN - and the server to respect this header and not cache.

So, I would F12->Network (f5) first file->Headers->Request Headers and see if you get back the same cache headers using your app_id and app_key.

This looks like a good description of the issues - Caching Tutorial for Web Authors and Webmasters

dev_rob · January 2, 2021, 1:21pm

The issue has resolved itself now.

I was getting the same results in two different environments (one at home and one in a cloud host), which would have pointed to a CDN or Varnish issue. But as it happens, it’s gone away now. Hopefully it won’t return.

nathanodong · January 8, 2021, 5:38pm

This happened to me last year too. Definitely a server-side caching issue that should be looked at.

dev_rob · January 19, 2021, 2:19pm

It’s reoccuring once more. I’d rather not do a workaround (like adding a random value to the query string). But it’s making my app unreliable. Is this is a ‘known issue’ requiring a workaround?

briantist · January 19, 2021, 3:21pm

Let’s ask @jamesevans ! #ping

jamesevans · January 20, 2021, 12:52pm

We have seen this in the past but not for quite a while. We’re doing a release this afternoon which should resolve it in the short term as we’ll be using a different environment.

We’ll then look at the current environment to see if there’s any issues in the Varnish cache that need sorting out.

Thanks,
James

dev_rob · January 20, 2021, 1:17pm

Super, thanks for the update.

dev_rob · January 28, 2021, 11:08pm

Unfortunately the issue has come back again, but in a slightly different way. Most requests were getting the stale response but about 1 in 5 times the latest response was received. I’ve added a workaround (a static parameter on the query string) which seems to be enough to force the response to update.

Shortcut · February 9, 2021, 3:24pm

@jamesevans It seems that this issue is back again.

Example: https://api.tfl.gov.uk/line/northern/Arrivals

→ data returned includes (note the 3-day old “timestamp”):

{
	"$type": "Tfl.Api.Presentation.Entities.Prediction, Tfl.Api.Presentation.Entities",
	"id": "-232023901",
	"operationType": 1,
	"vehicleId": "006",
	"naptanId": "940GZZLUMDN",
	"stationName": "Morden Underground Station",
	"lineId": "northern",
	"lineName": "Northern",
	"platformName": "Northbound - Platform 5",
	"bearing": "",
	"destinationNaptanId": "940GZZLUMDN",
	"destinationName": "Morden Underground Station",
	"timestamp": "2021-02-06T09:20:34.7025045Z",
	"timeToStation": 1026,
	"currentLocation": "Between Clapham North and Clapham Common",
	"towards": "Morden via CX",
	"expectedArrival": "2021-02-06T09:37:40Z",
	"timeToLive": "2021-02-06T09:37:40Z",
	"modeName": "tube",
	"timing": {
		"$type": "Tfl.Api.Presentation.Entities.PredictionTiming, Tfl.Api.Presentation.Entities",
		"countdownServerAdjustment": "00:00:00",
		"source": "0001-01-01T00:00:00",
		"insert": "0001-01-01T00:00:00",
		"read": "2021-02-06T09:20:54.967Z",
		"sent": "2021-02-06T09:20:34Z",
		"received": "0001-01-01T00:00:00"
	}
}

… with the response headers including:

x-cache: HIT
x-cache-hits:8968

However, if I change the URL by adding a query string, as follows:

https://api.tfl.gov.uk/line/northern/Arrivals?test=123

…then the data returned is fresh:

{
	"$type": "Tfl.Api.Presentation.Entities.Prediction, Tfl.Api.Presentation.Entities",
	"id": "-1300698428",
	"operationType": 1,
	"vehicleId": "103",
	"naptanId": "940GZZLUEUS",
	"stationName": "Euston Underground Station",
	"lineId": "northern",
	"lineName": "Northern",
	"platformName": "Northbound - Platform 3",
	"direction": "outbound",
	"bearing": "",
	"destinationNaptanId": "940GZZLUHBT",
	"destinationName": "High Barnet Underground Station",
	"timestamp": "2021-02-09T15:23:20.1861803Z",
	"timeToStation": 1152,
	"currentLocation": "Departed Oval",
	"towards": "High Barnet via Bank",
	"expectedArrival": "2021-02-09T15:42:32Z",
	"timeToLive": "2021-02-09T15:42:32Z",
	"modeName": "tube",
	"timing": {
		"$type": "Tfl.Api.Presentation.Entities.PredictionTiming, Tfl.Api.Presentation.Entities",
		"countdownServerAdjustment": "00:00:00",
		"source": "0001-01-01T00:00:00",
		"insert": "0001-01-01T00:00:00",
		"read": "2021-02-09T15:23:45.605Z",
		"sent": "2021-02-09T15:23:20Z",
		"received": "0001-01-01T00:00:00"
	}
}