Bus & River Predictions Unavailable


#1

All,

We’ve been informed of an issue that is currently affecting Countdown (bus & river) arrival predictions.

This means there are no bus or river predictions that will be returned from both the Unified API & legacy Countdown or shown on any physical displays at bus stops/stations.

It appears there is a networking issue that my colleagues are investigating as a high severity incident.

Our ref: SVC-4949

Thanks,
James Evans
Technology Service Operations


#2

I’ve been advised by my colleagues that services are recovering on displays around the network.

However, the Countdown API is still unavailable and that is being investigated.

Regards,
James


#3

All,

The Countdown API was back up and running at 21.23 last night.

We apologise for any inconvenience this may have caused you.

Cheers,

Dave


#4

Hi David

The legacy countdown API was taking at least 20 seconds to show a result this morning from around 7-8:30.

Lots of complaints from users.

Any idea what’s going on?


#5

Hi Jason,

I will raise this with colleagues who manage the countdown API. Hopefully we are able to get to the cause and resolve it ASAP.

Apologies that this has affected your customers.

Dave


#6

Hi,
I confirm what Jason said. Plus in the last 4/5 days it seems that the Countdown Api is having problems regularly in the morning rush hour, around 7/8.

Please fix this issue because users are complaining a lot.

Thank you
Giorgio


#7

Hi David

Were you able to find the cause of the issue. It seems to be continuing to affect all users during the busiest time of the morning. Complaints are in the hundreds.


#8

Hi Jason,
still having problems right now. Do you know what’s happening ?

Thank you
Giorgio


#9

Hi,

We were experiencing slow response rates from some API instances at approximately 0800 and restarted all instances to address.

We continue to monitor.

Thanks,

Mark.


#10

Hi,
we’re still experiencing the same problem today, is anyone working on this issue ? Any update please ?


#11

Hi,

Just to add my voice to this as well, the legacy API is taking along time if at all to respond and we’re getting alot of complaints! Is it being looked at?

Cheers
Shane


#12

Hi,

We’re restarting API instances and continue to investigate.

Thanks,

Mark.


#13

Given the problem roughly starts at 7.30 could you restart your instances at 7.31 please ? What’s the status of the investigation ?

Thank you


#14

I appreciate restarting API instances is not resolving this.

We are investigating the underlying issue as a top priority.

On behalf of TfL, I apologise the issues that this is causing you all.

Dave


#15

Hi Dave,
thank you for your update. I appreciate you’re investigating this issue but as this problem is happening for more than a week did you manage to find what’s causing it ? do you know when this will be fixed in production ?

Thank you


#16

It seems the system is misbehaving again.

Legacy and Unified API are both giving misleading results for my local route - E3.

I saw the ‘no arrivals in next 30 minutes’ message and the bus arrived 2 minutes later.

I checked Legacy data and it’s showing incorrect info for bus positions. There are also certainly more buses on the route than it is showing.

However the site is not advising any problem.


#17

@Dreq

The issue with the E3 was believed to be related to a Countdown release issue that was happening since Saturday but has since been resolved.

see this topic for more details.


#18

Hi, there’s still the excessive slow response issue going on. Why this top priority issue hasn’t been fixed yet ?
Thank you.


#19

@nakkore

Please see the update I’ve posted here: Daily Slow Performance on Countdown API

I’ll keep that updated with progress.

I’ll close this thread for now as we it’s become quite confused with a number of issues being discussed.

Thanks,
James


#20