NOW LIVE! Countdown API moving behind Cloudflare - NEW REVISED DATE: 11.00am Mon 25 Sept (not 21 Aug or 4 Sept)

[SarahLS]

[EDITED 25 Sept] This change is now LIVE! Please let us know (post below) if you are encountering any issues with Countdown at your end, or have any questions.

[EDITED 1 Sept] Due to some internal changes at our end, we are now unable to go ahead with this on Mon 4th Sept. We are very sorry for any inconvenience this may have caused.

REVISED DATE: if you are a Countdown API user, please note the NEW planned date/time for this change is 11.00am Monday 25th September 2023.

IMPORTANT Alert for Countdown API users

We will shortly be making some changes to how our users access the Countdown API, and your action will be required in order to continue to use this data.

If you are using the Countdown API, please read the information below about the changes and how you can test this in advance.

What are the Countdown API changes and why are they being actioned?

We have recently been experiencing severe load issues with our Countdown API. In order to improve the security and robustness of this feed and better manage demand, we will be moving the Countdown API behind Cloudflare. This should also improve overall performance for users.

We will be enforcing HTTPS with a minimum TLS version of 1.2. Users will need to ensure that the client they use to consume Countdown API feeds is compatible with TLS 1.2 or 1.3 protocol. If you are using an older client, this will no longer be compatible with our Countdown API feed (once we have rolled out the changes), and will need to be upgraded.

When will this change be introduced?
[EDITED 1 Sept] The new planned date/time for this change is: 11.00am Monday 25th September 2023.

**
What happens next?

Please follow the instructions below to test the new configuration ahead of rollout as soon as possible. If you encounter any issues with using the test domain, please reply to this post below.

How can I access the test?

We have set up a test domain: countdown-cf-test.tfl.gov.uk which has the configuration that will be implemented, and points to the live data. You can use this to test against, to allow you to understand the impact on your systems.

An example call using this domain:

https://countdown-cf-test.tfl.gov.uk/interfaces/ura/instant_V1?Stopcode2=490001144M&ReturnList=StopCode1,StopCode2,StopPointName,LineName,DestinationText,EstimatedTime,MessageUUID,MessageText,MessagePriority,MessageType,ExpireTime,DirectionID,VehicleID,RegistrationNumber

IMPORTANT: Once the changes have been applied to the main countdown domain (countdown.api.tfl.gov.uk), the test domain will be removed.

@jamesevans
@jwithers

[harry]

@SarahLS

Despite not knowingly using the Countdown API, I decided to test it “just in case”

I received this reply:

Browser is firefox 113.0.2 (64 bit)

[briantist]

@harry It’s not JSON. It’s just more like CSV with some start-and-end square brackets.

Also it’s a HTTP Digest. Digest - HTTP | MDN

[LeonByford]

Hi, just to clarify: the Countdown API returns data as JSON. However, rather than the entire response being a single JSON object, each line of the response is a JSON array.

Here is a Python example of how to consume the API:

import json
import requests

url = 'https://countdown-cf-test.tfl.gov.uk/interfaces/ura/instant_V1?Stopcode2=490001144M&ReturnList=StopCode1,StopCode2,StopPointName,LineName,DestinationText,EstimatedTime,MessageUUID,MessageText,MessagePriority,MessageType,ExpireTime,DirectionID,VehicleID,RegistrationNumber'
r = requests.get(url)
for line in r.iter_lines():
    if line:
        print(json.loads(line))

Therefore, the error message you see when querying the API in your browser is expected behaviour.

[Mulder]

This is a very tight timescale for an API which hasn’t changed in almost/over? a decade!! Are you sure that 21 August is achievable and have you considered the impact of what will happen when thousands of indie developers find out on the day that they can no longer access the API? 3 weeks for a change of this size seems very ambitious.

I will start pointing all my test environments to this URL.

[Mulder]

Please note that the URL in “How can I access the test?” doesn’t actually work as you forgot out the “tfl” subdomain in the URL. The link in the example works (as @harry pointed out) and the data is returned as expected.

I tested a bit this morning and all seems fine for now.

I will need some clarifications, because this is a fairly big change and I need to trust that TfL is managing this correctly:

1. To confirm, from 21 August (what time???), instead of using http://countdown.api.tfl.gov.uk, we will only change h
   ttp to https and it will become https://countdown.api.tfl.gov.uk - as long as we have a minimum TLS version of 1.2, it will work. (I’ve confirmed that all my services comply with this).
2. From 21 August, the old http version of the API will stop working. What error will we receive? Is there a way to test what will happen with the old service? It would be useful so that I can be proactive about errors users may see in production.
3. There is NO RISK that this change will be postponed. The change is three weeks away and I will need to script everything for the switchover. The last thing I would want to do is to prepare for such a big change and then have to revert or roll back everything. I would like to have confirmation that this change will definitely go ahead and it will remain in place if the technology works (e.g. what will happen if the switchover is successful, but you receive hundreds of complaints from developers who were unaware of this???). I do not want to do a huge amount of work, only for it to be undone because someone else wasn’t prepared.
4. Will there be any dual running period where both https and http will continue to work?

I am really looking forward to this change, because it will hopefully resolve some of the issues we have recently seen with countdown. I am happy with the tight timescales, as long as I can have guarantees from TfL that everything will go ahead.

For information so that you also understand the downstream impact, I am an app developer who will have to ensure that every single user of mine updates to the latest app version within the next 3 weeks or it will break, so I will have to communicate this change out as soon as possible (this is, I have to update them to a version of the app which is not even out yet and I have no guarantees that I won’t update the app only for the change to be reverted). Given that in the last 7 days, over 10% of Android users are using a version of the app which is more than 3 months old (and new versions are available almost weekly), it is not a small task and I will be annoying a lot of users to make sure we get everyone working.

[Mulder]

Another question (I am sure it won’t be the last as I am going through the change and seeing the impact as I go):

5. Can I safely convert a portion of my users to use the test domain in production for the next few weeks or is that a bad idea? I fully understand that the test domain will be turned off from 21 August onwards, but I would also like to get an idea of any issues with the new service before 21 August, rather than after.

[LeonByford]

Hi, thank you for your feedback.

@Mulder

This is a very tight timescale for an API which hasn’t changed in almost/over? a decade!! Are you sure that 21 August is achievable and have you considered the impact of what will happen when thousands of indie developers find out on the day that they can no longer access the API? 3 weeks for a change of this size seems very ambitious.

I will start pointing all my test environments to this URL.

This migration is necessary to address the performance issues you have noticed recently. Therefore, although we appreciate it puts a fair bit of pressure on open data users, we do need to move quite quickly with this.

In addition to this forum post, we are emailing open data users to try to give as much forewarning as possible.

If there is anything we can do to support migration efforts, please let us know.

@Mulder

Please note that the URL in “How can I access the test?” doesn’t actually work as you forgot out the “tfl” subdomain in the URL. The link in the example works (as @harry pointed out) and the data is returned as expected.

Thank you for reporting this. I have corrected the hostname in the first post.

@Mulder

To confirm, from 21 August (what time???), instead of using http://countdown.api.tfl.gov.uk, we will only change h
ttp to https and it will become https://countdown.api.tfl.gov.uk - as long as we have a minimum TLS version of 1.2, it will work. (I’ve confirmed that all my services comply with this).

This should work. I’m not sure we can provide an exact time for this change, but if you want to ensure there is no downtime, you might like to do the following:

• At some point before 21st August the migration happens, make a change so you still request http but follow redirects. Once we make the change, any http requests will result in a redirecting status code such as 301 or 307 with a redirect to the https URL. Therefore the migration should be automatic.
• At some point after the migration, ideally you would change the URL to request https specifically.

@Mulder

From 21 August, the old http version of the API will stop working. What error will we receive? Is there a way to test what will happen with the old service? It would be useful so that I can be proactive about errors users may see in production.

It will behave as countdown-cf-test currently behaves when you send it an http request, for example:
http://countdown-cf-test.tfl.gov.uk/interfaces/ura/instant_V1?Stopcode2=490001144M&ReturnList=StopCode1,StopCode2,StopPointName,LineName,DestinationText,EstimatedTime,MessageUUID,MessageText,MessagePriority,MessageType,ExpireTime,DirectionID,VehicleID,RegistrationNumber
In other words, an HTTP redirect to the https URL.

@Mulder

There is NO RISK that this change will be postponed. The change is three weeks away and I will need to script everything for the switchover. The last thing I would want to do is to prepare for such a big change and then have to revert or roll back everything. I would like to have confirmation that this change will definitely go ahead and it will remain in place if the technology works (e.g. what will happen if the switchover is successful, but you receive hundreds of complaints from developers who were unaware of this???). I do not want to do a huge amount of work, only for it to be undone because someone else wasn’t prepared.

It is unlikely that this change would be postponed or reverted, but we cannot give a complete guarantee of this. That said, if you take the approach I mentioned previously with following the redirect, this should minimise the amount of work for yourself and you’ll be prepared for those kinds of scenarios.

@Mulder

Will there be any dual running period where both https and http will continue to work?

There will be no dual running. After the migration, http requests will return a redirect.

@Mulder

Can I safely convert a portion of my users to use the test domain in production for the next few weeks or is that a bad idea? I fully understand that the test domain will be turned off from 21 August onwards, but I would also like to get an idea of any issues with the new service before 21 August, rather than after.

The test feed is not intended for production use, and therefore I would not recommend doing this.

[Mulder]

Hi Leon, thank you so much for your very detailed response to all of my questions (especially on a Sunday!!!).

I wasn’t aware of the redirect and this will probably solve 99% of potential problems, since it just works for me without having to change anything (of course, except for the URL in testing). (This is what I was after for the dual running, I know that the response is technically a bit different, but in reality, the same request will still give the same response, so I think for the majority of users, no change will ultimately have minor impact - for competitive purposes, I was secretly hoping that everything would break, but it’s risky for me too).

Thanks for all the work and I’m looking forward to a robust and performant countdown API.

A follow-up question is if there is any plans to shut down the countdown API in future or force developers to use the unified API instead? I’ve been postponing it for years since I’ve keyed my whole app on StopCode1 and I would need to migrate to using StopCode2 first, since StopCode1 isn’t very useful on the unified API. (If you made it available tomorrow, I would switch immediately).

[LeonByford]

You’re very welcome.

The Countdown API isn’t going away anytime soon. The Unified API actually gets its arrivals data from the Countdown API. Therefore we couldn’t shut down the Countdown API without also breaking the Unified API (and some of our other systems).

If this changes in future, it will be communicated on this forum and we will try to give as much notice as possible.

By the way, important note: although the redirect means that some applications will continue to work without any changes required (since http will redirect to https), there will be applications where redirects are not automatically followed. Therefore, I strongly recommend testing with the test feed to check the behaviour of your system. You may need to configure your requests library to follow redirects, or implement this yourself. Please let us know if you require any assistance with this.

[Bruno]

Hi. I’m trying to use https://countdown.api.tfl.gov.uk right now and it’s not working. Are we going to be able to start using the HTTPS version of the current URL before the deadline?

[LeonByford]

Hi. No, this will not be available until 21st August the migration happens.

[Mulder]

@LeonByford

The test feed is not intended for production use, and therefore I would not recommend doing this.

I would like to double-check - whilst it’s not your recommendation, is there any reason not to put some traffic through the test version (with all the caveats that it’s a test version)? (E.g. if it is an exact replica of what you intend to deploy on 21 August, but with no support when things go wrong, I’m still happy to try and move traffic; if it’s not going to make things better right now, then I won’t).

The countdown feed is so incredibly broken today (average response time of over 22 seconds for around 5000 samples for today so far!!!), I am desperate to try anything to make all the complaints stop.

My reasoning is that the countdown feed is currently in a state which is not production-ready; so moving onto something else which is considered not to be production-ready, but performing better, it’s better than what I have to deal with right now.

(My testing environment, which uses the new feed doesn’t seem to be suffering so much - I get the odd timeout or slow response, but it’s not nearly as bad as the current API).

The sad state of affairs for http://countdown.api.tfl.gov.uk/** - response time for the last 90 days, despite traffic dropping due to school holidays:

[LeonByford]

Thanks for your question. As you say, we wouldn’t be able to provide support for the test feed to the same level as the production feed. Also, please remember that the test feed will be removed after the migration on 21st August.

[misar]

Your email confused me and this blog has not resolved my question.
I have been using https://api.tfl.gov.uk for years and that is working fine right now.
Are you saying that from 21 August I can use my code unchanged provided I change that URL to https://countdown.api.tfl.gov.uk?

[LeonByford]

Hi, apologies for the confusion.

To clarify, this only affects people who are already using countdown.api.tfl.gov.uk. If you are only using api.tfl.gov.uk, this will not affect you and you can safely ignore our email.

Please let us know if you have any further questions.

[misar]

Thank you for the quick reply.
That is very good news!

[MikeHampton]

We are currently using an old interface that is a stream called “Live Bus & River Bus Arrivals API”.

We do get this by using countdown.api.tfl.gov.uk

Will that stream be affected by the changes you describe regarding Countdown API moving behind Cloudflare?

[LeonByford]

Hello and welcome to the forum.

The Live Bus & River Bus Arrivals API is another name for the Countdown API. This (including the stream_V1 endpoint, which I assume you’re using) will be affected by this change.

The stream will be served over HTTPS. You can check the behaviour with our test hostname countdown-cf-test.tfl.gov.uk.

Please let us know if you have any further questions.

[Mulder]

Hello, can I confirm:

• This change is still going ahead on 21 August as planned
• Will you send an update to the community once the change has been made?

I would like to closely monitor things to be sure that it didn’t break anything.

Thanks!