Buses that disappear from predictions API

Thanks James. I look forward to your findings.

Harry

1 Like

@jamesevans Thanks from me too.

@harry

Given that I worked on GPS when it was a secret military thing… I guess I can’t make any additional comments about the accuracy statement.

From what I recall being told by TfL people, the bus drivers unions didn’t want bus locations to be put out into the public because they feared that criminals would use the data to rob the bus drivers… whilst I can’t see any logic to this at all (given the general visibility of said red machines) it does seem to mean that TfL have the live data in iBus but don’t push it to the public.

It’s a real shame because this kind of thing gives the travelling public confidence in the viability/reliability of public transport. The Liz Line, for example, has message like “the train is between Maryland and Stratford” which is very good.

So yes, all the coding attempts to work out where a bus to show them on a map - such as Live London bus map - are rather hampered by roundabouts and gyratory systems.

Having written code myself that show the live positions of trains on diagrams, I can confirm that it’s hard to get right and you should not make assumptions about events that may not have happened…

I do the same … on Greater Anglia, it takes a few minutes longer for my train to go from the terminus to my local station than it does for me to walk from home to the station. So when I can see that the train has left the terminus, I have a lot more confidence that my train is likely to arrive on time than just seeing “On Time” but hasn’t left which, for all I know only means “we haven’t cancelled it yet, but we still might”.

As for the bus drivers – in the days when they used to carry cash (and even walk it back to the garage, after a change of drivers), there was probably good reason to keep their location a little bit vague. Though as you say, those brightly painted red objects aren’t easy to disguise.

My suspicion is that it has more to do with not wanting anybody other than the controllers to be able to systematically monitor whether the bus performance was meeting its targets – and if that’s genuinely not the real reason, then TFL could at least release data of actual arrival time into history a few minutes after each arrival.

1 Like

One of the arguments that some used against releasing bus working timetables was that it made it easier for someone who had had an argument with the driver to find out where and when they might “re-engage” with the driver. I’m not aware of that ever having happened (and you might think that anyone capable of working through the duties etc on a working document would be unlikely to be interested in assault). I can however see that at some isolated termini a driver might feel isolated and vulnerable.

I suppose actual arrival time (except at the last stop) could be extracted by observing when the next stop moves on by one.

Am I dreaming things or did I read that actual location would be supplied to the new national bus system?

@jamesevans

Silly question: does iBus stop predicting if the engine is switched off at the terminus?

Does iBus, perhaps, rely on a battery backup to maintain its data when the engine is not running?

Most of the buses on the 97 route are a similar age … so probably last had their batteries fitted at the same time, and might therefore be due for replacement.

@harry As motor vehicles, the power for the 12V DC services are provided by the diesel/petrol engine that trickle charges the led-acid accumulators (batteries).

Without such power the vehicle won’t start as it requires a large power draw to turn the alternator to get the engine moving.

The amount of power used to connect to the 5V DC services such as iBus are trivial: if the bus won’t start it will get topped up (or a new battery).

But we have seen reports of police cars having to leave their engines running, even sometimes when parked for hours on end. The excuse seems that the companies that fitted their computers didn’t have the sense to fit battery backup to their computers (or equip them with non-volatile ram). But a police car on an emergency call can’t afford to wait several minutes while their computers reboot.

That said, there’s obviously a lot more space on a bus where bigger batteries can be fitted. Though I’m still surprised if they use the same batteries to provide back up power to the computer as they do to start the engine. Buses might be very different from cars, but when you start a car, the battery voltage drops temporarily to just a couple of volts.

Total coincidence, but today I travelled on one of those “invisible” buses.

At Chingford station, the countdown display showed the first 97 as coming in 11 minutes. I guessed, from the previous tests, that the first 97 would come along in about 4 minutes … and it did. With the countdown at that time still showing 9 minutes for the next 97 bus.

I chose to trust that this big red thing with 97 on the front was actually a real bus even though the countdown display confirmed it was really only a figment of my imagination. Hence I risked getting on it.

Not only that, but the iBus system was obviously working – at least, to the extent that the bus knew where it was, displayed the correct time, and announced the correct stops as it approached them.

The non-existent bus arrived at its fourth stop (stop Q, the one I’ve mostly been watching because that’s where I first noticed the problem.

Yes, the bus (LX09FYT) definitely knew exactly where it was:

But the stop it was at was not predicting that bus, or indeed any other, to be alongside for another 12 minutes :

image

Did anything interesting come from that?

Other activities have prevented me following this up for the past month, but I’m still seeing exactly the same problem almost every time I look at the predictions for that fourth stop (Q).

An example from a few minutes ago:

2022-05-12 08_25_12-busTimes — Mozilla Firefox

No buses predicted at the 4th stop for the next 20 minutes, yet simultaneously at the third stop:

2022-05-12 08_24_58-

there’s a bus LX09ABN which apparently is not predicted to ever reach that fourth stop.

1 Like

@jamesevans

Doing some further tests, I find the problem isn’t only with route 97. Other buses starting from Chingford Station also have the problem.

313 is one of them. On many occasions I’ve gone to 490005182N at a time when no buses were predicted for 25 minutes or more but the timetable has said there will be one in 5 minutes. So I’ve gone to the stop and a bus has arrived within a minute or two of the scheduled time.

So I did a specific tests on route 313 (2nd September)

at 9:25, stop 490005182N was predicting a bus LTZ1180 in 16 minutes:
image

That continued to be predicted each minute up to 9:36:
image

but disappeared from predictions at 9:37:
image

Until 9:43 predictions continued to show the bus would not be coming:
image

But at 9:44 suddenly it reappears with only 1 minute to go:
image

Meanwhile, watch what happens to LTZ1001 due in 16 minutes: it continues to be predicted every minute, up to 9:53:
image

And then disappears at 9:54:
image

That bus also reappears with just 1 minute to go:
image

That pattern continued for subsequent buses on the route:

  • LTZ1181 disappeared at 09:11, with predicted 8 minutes to arrival, and reappeared at 9:20, with 1 minute to go

  • LTZ1182 disappeared at 09:32, with predicted 10 minutes to arrival, and reappeared at 9:41, with 2 minutes to go

Enough of that, let’s look back at what might have been happening when these buses disappear.

At 09:27, LTZ1180 was still on its way to the terminus, to arrive 3 minutes later (or at least, that’s what the API thought) :
image

and at that time it was expected to leave after just a 5 minute break:
image

3 minutes later though, it still seemed to be 3 minutes away from terminus:
image
but was still thought to be at its first outbound stop at 09:34:48 and was still showing predictions for the outbound stops:
image

At 9:35, predictions were still being shown on the assumption of leaving at 09:34:48:
image

It’s perhaps at this time that “something or somebody” decided the incoming bus had not arrived late and therefore wasn’t going to leave on time. But instead of adding several minutes to the outbound predictions the bus was removed from predictions altogether.
image

A few minutes later, it seems that the same “something or somebody” failed to report that the bus was ready to leave, and apparently iBus isn’t yet sufficiently intelligent to deduce from the engine being switched back on, or its GPS location no longer being that of the terminus, is a strong indication that predictions should be reinstated.

It seems, furthermore, that even after leaving the terminus, a bus can travel a whole 3 stops (5 minutes) before iBus informs the API that it should restart predictions.

But anyway, unless something very serious has happened (like the bus getting a puncture, or the driver reporting sick, surely it’s obvious that the bus will usually leave a bit late rather than not at all. So its predictions should be delayed by a few minutes rather than falsely pretending that no bus would be coming at all.


This morning I also monitored the 212 which starts from Chingford

It looked good at first:
image

But 1 minute later LF20XLP disappeared with 5 minutes to go:
image

Then so did LF20XLA:
image
And so did LF20 XMC:
image

Unlike the 313, the missing buses never reappeared in any predictions for the monitored stop, but they did appear in predictions further along the route, so again these buses are managing to escape from the terminus without being picked up by the API for a long time thereafter.

But to a person looking at predictions, it will usually appear that the next bus is 10-15 minutes away, even though the actual bus will arrive in 1 to 8 minutes.

And here’s an interesting prediction for a 179 (another route starting from Chingford)

Retrieved at 08:17:05, apparently, this bus is predicted to arrive at its first stop 10 seconds ago. OK, that’s not unreasonable for a prediction.

Except that:

  • it’s going to skip stops 2 to 6 and arrive right just 10 seconds later at stop 7
  • and then on to stop 8 in a plausible 51 seconds
  • then it’s going to realise it missed some stops and double back to stop 2, a feat that it can manage in just 20 seconds
  • but it doesn’t think stops 3 to 6 are worth bothering with, so next is stop 9

As with the other routes monitored, that bus wasn’t being included ini the individual stop predictions for stops 3 to 6.

I’d like to have had some earlier history, but this popped up at the precise instant I started the monitor. But I’ll leave it running for a few hours and see it the same happens with any other buses on that route.

Meanwhile, it’s fairly obvious that the predictions for that single bus weren’t generated from a single plausible set of data and seem to relate to whatever generates the predictions not reliably knowing the current location of the bus.

1 Like

@harry This sounds 100% like the 97 going Northwards from Stratford City when I used to try and catch it.

Keep up the good work!

OK, lets look each minute at the predictions for Stop 4 (GA, 490005183E): Balgonie Road
image
image
image

Where did SN63JVZ go?

Its vehicle prediction showed it as arriving at stop 4 at 09:10:49

But then no more:
image

until 09:12:11:

OK, so it’s now showing as being a couple of minutes later than previously expected.
But what happened between 09:04 and 09:12 to make the API think it should be telling everybody that this bus would not arrive at all?

That’s exactly the same as the 97 problem.

And it’s repeated with LX11BGU:
image
which disappears 1 minute later:
image
as does LX61DAA a couple of minutes later:
image
Then a new bus appears, but it’s not apparently coming for 27 minutes (despite this being a regular 7 minute service)
That’s still the only bus showing at 09:21:
image
when suddenly BGU reappears, with 0 minutes notice:
image
That’s not much help if you live 2 minutes from that bus stop, but were told there would be no bus for 25 minutes. It’s coming only a few seconds later than originally predicted ten minutes earler. But again, we need to find out why the API thought from 09:12 to 09:21 that it would not be coming at all.

1 Like

@jamesevans

It’s now something like 9 months since I raised this problem but it appears not to have even been investigated. There is clearly something completely wrong with either the data coming from buses leaving the Chingford terminus or with what TFL is doing with that data when it presents it via the API and the countdown screens.

I initially thought it was a problem specific to 97, I’ve quoted evidence that it applies to other services, from different operators, on routes starting from Chingford. That includes at least 212, 197 and 313.

The predictions API, as currently implemented, is clearly completely unfit for its purpose, at least in respect of the first half dozen stops of buses starting from Chingford.

Could I please ask for a specific New Years Resolution, to get this unnecessary problem fully and properly investigated and fixed.

1 Like

hi @harry

Sorry for the delay in response to you.

The issue appears to be further down the stack than we can look at in the API. The API itself seems to be processing the data we receive from iBus/Countdown correctly, so we raised this with our colleagues who look after those systems.

This appears to be related to the iBus system’s use of 2G mobile network infrastructure. This is still under investigation with our colleagues in that area but the working theory is that coverage on 2G has been degraded across London by the cellular networks so that 4G & 5G services can be provisioned on existing towers. We have been doing analysis of our live vehicle tracking and working out the areas affected and the reliability of the data therein.

Again, I’m hoping to catch up with my colleagues on this matter this week and I’ll update the thread when I have some more information or updates from them.

Thanks,
James

1 Like

Thanks James.

Lack of 2G in the area seems like a fairly likely cause, but I’m surprised that iBus doesn’t have the capability of detecting lost transmissions in some way.

And if 2G is being deliberately degraded rather than just exhibiting occasional short term temporary faults, it’s strange that the networks haven’t confessed that to ofcom. Today it still thinks all four networks are providing OK voice coverage at Chingford Bus Station:

Of course, voice doesn’t mean 2G only and ofcom really ought to be providing separate columns for 2G and 3G. So I also checked at https://www.signalchecker.co.uk/ which has separate columns for “voice” and 3G, but that too claims good “non-3G” voice coverage indoors and outdoors for all four networks:

I also checked o2 coverage at https://status.o2.co.uk/ which specifically claims good 2G coverageindoors and outdoors (the blue covering the bus station (A) and all of the stops that aren’t getting proper predictions):

image

o2’s status checker does admit:
image
but it says that for all services, not specifically 2G.

Anyway, good to see it’s being looked into. I’m in the area about twice a week so I’ll quickly notice if any buses actually start being predicted to be less than 20 minutes away.

@jamesevans

Has this been fixed? Several times in the last few days I’ve seen predictions in the Chingford area that for the first time in well over a year are properly showing buses arriving in the next 10 minutes. It seems to be a recent change – it was certainly as wrong as ever just a few weeks ago.

Was it the 2G problem, or something else?

Further checking shows the problem is not completely fixed. It seems that although the first bus on any route is now (usually) shown in the predictions, the second bus on the same route is often missing (but the third and fourth are usually correctly shown.

And yesterday I boarded a 97 bus that I spotted coming round the corner, despite the predictions showing nothing coming for 9 minutes:

image

Yet here it is at 15:10

image

and that was almost exactly when it had been predicted to arrive a few minutes earlier, but had mysteriously been dropped from the predictions:

image

1 Like

@harry Indeed… I’ve been moaning about this exact point since I moved to E20 in 2013. :grin:

Thank you for your very detailed investigation into this. I am seeing this with several users where buses keep on disappearing and people are no longer trusting the system because buses just appear and disappear randomly. It seems to be increasing steadily as I am getting more and more complaints and I am unable to do such a thorough investigation like @harry did. Thank you!

1 Like