Buses that disappear from predictions API

[harry]

Maybe I spoke too soon. Here’s a disappearing prediction from the Stratford end of the route:

That ties in with the vehicle prediction at that time:

So, let’s watch LX08FZE and remember that it has been predicted at stop 3 for 09:50

At 09:22 it’s still predicted:

But at 09:23 it is mysteriously gone … and FZE is apparently going nowhere:

and now we’re expecting a 24 minute service gap:

This, remember, is on a “frequent 7 minute service” where there seem to be plenty of buses waiting in the terminus.

The predicted gap continues until AAO has passed:

but still with no prediction to plug the gap, until

So, FZE is coming anyway … it is now predicted to be a couple of minutes later than its previous prediction. But why did it disappear altogether from the prediction, instead of simply having it’s time revised?

[briantist]

@harry

Do these predictions really come solely from iBus?

I’ve been shown the iBus at various points by TfL people and it seems that the thing that people call “iBus” comprises of the ability to:

• Communicate the GPS positions of the buses to the control centre. This needs an external aerial on the bus to ensure enough satellites are received to get accurate calculations;
• Trigger the “the next stop is…” messages within the bus;
• Inform the driver of how far ahead/behind the timetable the bus is at any given time and also trigger “bus is waiting” announcements;
• provide “control” with the live data to allow them to make short-running decisions;
• communicate the upcoming bus data to the SMS-controlled at-stop-screens.
• provide external APIs that contain the live position and rest-of-run stop predications. This was originally done though an HTTP digest that just provides changes to predications because the on-stop displays are told the to-the-second times for the arrivals but display and work out their own minutes countdown.

So, yes, the iBus system expects buses to run as per the timetables and TfL expects that the bus operators run the buses to the timetable as per their contracts!

It’s actually down the contractors to run their buses to the timetable and to sort out the staffing issues that arise around this.

[harry]

Here’s a particularly nonsensical prediction from the 97 route this morning.

That’s right, the API thinks FZD a bus will leave the terminus in 7 minutes, pick up passengers at the first stop 1 minute later … and then keep them hostage while it skips the next 15 stops …

Here’s the actual data returned by https://api.tfl.gov.uk/Vehicle/LX09FZD/Arrivals a little after that was seen:

And, as is often the case, no other predictions for the next 20 minutes at the 4th stop:

That problem was ongoing:

Finally the bus seems to have left the station …

but is still not predicted to arrive at stop 4

and nor, apparently, is ZHC which was predicted to arrive there in my earlier screenshot
But that’s now going only as far as Stop 1:

And FZD still thinks it’s only going to stop 15:


Oh, FZD has had a change of mind and it’s now going to call at stops 4 onwards. I guess the hostage passengers rebelled and demanded to be allowed to alight:

And FZD is now going to call at the subsequent stops too:

But still, CHC is going nowhere much:

Meanwhile, I’ve checked route 279 again (with a similar frequency of about 7 minutes at peak periods) for several hours and nothing similar regularly happens there, so there’s got to be a difference of some sort with the way that route 97, specifically (and maybe others, but certainly not all routes od that frequency) is predicted.

@jamesevans

I’m not sure if it’s your job or somebody else, but the predictions on route 97 clearly aren’t fit for their purpose.

Is there, surely, somebody who can investigate whether it’s the bus equipment, its programming, its drivers, controllers on the route, equipment at the terminus at each end, or some other cause that’s making such a mess of the predictions … ???

[harry]

Update:

CHC finally reappeared in the vehicle prediction:

but did not at any time reappear in the Stop 4 prediction:

which once again lost yet another previously-predicted bus TGU:

And that’s precisely what happens at stop 4 for much of the day … there are rarely any buses predicted less than 20 minutes away.

[harry]

@briantist

provide external APIs that contain the live position and rest-of-run stop predications.

Live position, if iBus (London) - Wikipedia is to be believed, should be accurate to 10 metres even if the bus cannot receive GPS at the time.

Does that API exist as open data? Or even at all?

I guess it doesn’t, because https://api.tfl.gov.uk/Vehicle/${vehicle}/Arrivals would be the obvious place to put a vehcile’s current location, but it’s not there.

And if you look at Live London bus map you can see there’s a lot of guesswork going on, because the author can’t accurately determine the current position of the bus or even whether it has left the terminus,

This gives slightly hilarious results of a buses apparently “reversing backwards towards the terminus” to correct for a bus that was previously predicted to be arriving at stop 1, but is subsequently found not to have yet left the depot.

[jamesevans]

Hi @harry

I’m taking a look into the 97.

I’ll let you know what I find, but it’s likely to be Monday before I can get an answer.

Thanks,
James

[harry]

@jamesevans

it’s likely to be Monday before I can get an answer.

Thanks James. I look forward to your findings.

Harry

[briantist]

@jamesevans Thanks from me too.

@harry

Given that I worked on GPS when it was a secret military thing… I guess I can’t make any additional comments about the accuracy statement.

From what I recall being told by TfL people, the bus drivers unions didn’t want bus locations to be put out into the public because they feared that criminals would use the data to rob the bus drivers… whilst I can’t see any logic to this at all (given the general visibility of said red machines) it does seem to mean that TfL have the live data in iBus but don’t push it to the public.

It’s a real shame because this kind of thing gives the travelling public confidence in the viability/reliability of public transport. The Liz Line, for example, has message like “the train is between Maryland and Stratford” which is very good.

So yes, all the coding attempts to work out where a bus to show them on a map - such as Live London bus map - are rather hampered by roundabouts and gyratory systems.

Having written code myself that show the live positions of trains on diagrams, I can confirm that it’s hard to get right and you should not make assumptions about events that may not have happened…

[harry]

@briantist

The Liz Line, for example, has message like “the train is between Maryland and Stratford” which is very good.

I do the same … on Greater Anglia, it takes a few minutes longer for my train to go from the terminus to my local station than it does for me to walk from home to the station. So when I can see that the train has left the terminus, I have a lot more confidence that my train is likely to arrive on time than just seeing “On Time” but hasn’t left which, for all I know only means “we haven’t cancelled it yet, but we still might”.

As for the bus drivers – in the days when they used to carry cash (and even walk it back to the garage, after a change of drivers), there was probably good reason to keep their location a little bit vague. Though as you say, those brightly painted red objects aren’t easy to disguise.

My suspicion is that it has more to do with not wanting anybody other than the controllers to be able to systematically monitor whether the bus performance was meeting its targets – and if that’s genuinely not the real reason, then TFL could at least release data of actual arrival time into history a few minutes after each arrival.

[mjcarchive]

One of the arguments that some used against releasing bus working timetables was that it made it easier for someone who had had an argument with the driver to find out where and when they might “re-engage” with the driver. I’m not aware of that ever having happened (and you might think that anyone capable of working through the duties etc on a working document would be unlikely to be interested in assault). I can however see that at some isolated termini a driver might feel isolated and vulnerable.

I suppose actual arrival time (except at the last stop) could be extracted by observing when the next stop moves on by one.

Am I dreaming things or did I read that actual location would be supplied to the new national bus system?

[harry]

@jamesevans

Silly question: does iBus stop predicting if the engine is switched off at the terminus?

Does iBus, perhaps, rely on a battery backup to maintain its data when the engine is not running?

Most of the buses on the 97 route are a similar age … so probably last had their batteries fitted at the same time, and might therefore be due for replacement.

[briantist]

@harry As motor vehicles, the power for the 12V DC services are provided by the diesel/petrol engine that trickle charges the led-acid accumulators (batteries).

Without such power the vehicle won’t start as it requires a large power draw to turn the alternator to get the engine moving.

The amount of power used to connect to the 5V DC services such as iBus are trivial: if the bus won’t start it will get topped up (or a new battery).

[harry]

But we have seen reports of police cars having to leave their engines running, even sometimes when parked for hours on end. The excuse seems that the companies that fitted their computers didn’t have the sense to fit battery backup to their computers (or equip them with non-volatile ram). But a police car on an emergency call can’t afford to wait several minutes while their computers reboot.

That said, there’s obviously a lot more space on a bus where bigger batteries can be fitted. Though I’m still surprised if they use the same batteries to provide back up power to the computer as they do to start the engine. Buses might be very different from cars, but when you start a car, the battery voltage drops temporarily to just a couple of volts.

[harry]

Total coincidence, but today I travelled on one of those “invisible” buses.

At Chingford station, the countdown display showed the first 97 as coming in 11 minutes. I guessed, from the previous tests, that the first 97 would come along in about 4 minutes … and it did. With the countdown at that time still showing 9 minutes for the next 97 bus.

I chose to trust that this big red thing with 97 on the front was actually a real bus even though the countdown display confirmed it was really only a figment of my imagination. Hence I risked getting on it.

Not only that, but the iBus system was obviously working – at least, to the extent that the bus knew where it was, displayed the correct time, and announced the correct stops as it approached them.

The non-existent bus arrived at its fourth stop (stop Q, the one I’ve mostly been watching because that’s where I first noticed the problem.

Yes, the bus (LX09FYT) definitely knew exactly where it was:

But the stop it was at was not predicting that bus, or indeed any other, to be alongside for another 12 minutes :

[harry]

@jamesevans

I’m taking a look into the 97.

Did anything interesting come from that?

Other activities have prevented me following this up for the past month, but I’m still seeing exactly the same problem almost every time I look at the predictions for that fourth stop (Q).

An example from a few minutes ago:

No buses predicted at the 4th stop for the next 20 minutes, yet simultaneously at the third stop:

there’s a bus LX09ABN which apparently is not predicted to ever reach that fourth stop.

[harry]

@jamesevans

Doing some further tests, I find the problem isn’t only with route 97. Other buses starting from Chingford Station also have the problem.

313 is one of them. On many occasions I’ve gone to 490005182N at a time when no buses were predicted for 25 minutes or more but the timetable has said there will be one in 5 minutes. So I’ve gone to the stop and a bus has arrived within a minute or two of the scheduled time.

So I did a specific tests on route 313 (2nd September)

at 9:25, stop 490005182N was predicting a bus LTZ1180 in 16 minutes:

That continued to be predicted each minute up to 9:36:

but disappeared from predictions at 9:37:

Until 9:43 predictions continued to show the bus would not be coming:

But at 9:44 suddenly it reappears with only 1 minute to go:

Meanwhile, watch what happens to LTZ1001 due in 16 minutes: it continues to be predicted every minute, up to 9:53:

And then disappears at 9:54:

That bus also reappears with just 1 minute to go:

That pattern continued for subsequent buses on the route:

• LTZ1181 disappeared at 09:11, with predicted 8 minutes to arrival, and reappeared at 9:20, with 1 minute to go

• LTZ1182 disappeared at 09:32, with predicted 10 minutes to arrival, and reappeared at 9:41, with 2 minutes to go

Enough of that, let’s look back at what might have been happening when these buses disappear.

At 09:27, LTZ1180 was still on its way to the terminus, to arrive 3 minutes later (or at least, that’s what the API thought) :

and at that time it was expected to leave after just a 5 minute break:

3 minutes later though, it still seemed to be 3 minutes away from terminus:

but was still thought to be at its first outbound stop at 09:34:48 and was still showing predictions for the outbound stops:

At 9:35, predictions were still being shown on the assumption of leaving at 09:34:48:

It’s perhaps at this time that “something or somebody” decided the incoming bus had not arrived late and therefore wasn’t going to leave on time. But instead of adding several minutes to the outbound predictions the bus was removed from predictions altogether.

A few minutes later, it seems that the same “something or somebody” failed to report that the bus was ready to leave, and apparently iBus isn’t yet sufficiently intelligent to deduce from the engine being switched back on, or its GPS location no longer being that of the terminus, is a strong indication that predictions should be reinstated.

It seems, furthermore, that even after leaving the terminus, a bus can travel a whole 3 stops (5 minutes) before iBus informs the API that it should restart predictions.

But anyway, unless something very serious has happened (like the bus getting a puncture, or the driver reporting sick, surely it’s obvious that the bus will usually leave a bit late rather than not at all. So its predictions should be delayed by a few minutes rather than falsely pretending that no bus would be coming at all.

---

This morning I also monitored the 212 which starts from Chingford

It looked good at first:

But 1 minute later LF20XLP disappeared with 5 minutes to go:

Then so did LF20XLA:

And so did LF20 XMC:

Unlike the 313, the missing buses never reappeared in any predictions for the monitored stop, but they did appear in predictions further along the route, so again these buses are managing to escape from the terminus without being picked up by the API for a long time thereafter.

But to a person looking at predictions, it will usually appear that the next bus is 10-15 minutes away, even though the actual bus will arrive in 1 to 8 minutes.

[harry]

And here’s an interesting prediction for a 179 (another route starting from Chingford)

Retrieved at 08:17:05, apparently, this bus is predicted to arrive at its first stop 10 seconds ago. OK, that’s not unreasonable for a prediction.

Except that:

• it’s going to skip stops 2 to 6 and arrive right just 10 seconds later at stop 7
• and then on to stop 8 in a plausible 51 seconds
• then it’s going to realise it missed some stops and double back to stop 2, a feat that it can manage in just 20 seconds
• but it doesn’t think stops 3 to 6 are worth bothering with, so next is stop 9

As with the other routes monitored, that bus wasn’t being included ini the individual stop predictions for stops 3 to 6.

I’d like to have had some earlier history, but this popped up at the precise instant I started the monitor. But I’ll leave it running for a few hours and see it the same happens with any other buses on that route.

Meanwhile, it’s fairly obvious that the predictions for that single bus weren’t generated from a single plausible set of data and seem to relate to whatever generates the predictions not reliably knowing the current location of the bus.

[briantist]

@harry This sounds 100% like the 97 going Northwards from Stratford City when I used to try and catch it.

Keep up the good work!

[harry]

OK, lets look each minute at the predictions for Stop 4 (GA, 490005183E): Balgonie Road

Where did SN63JVZ go?

Its vehicle prediction showed it as arriving at stop 4 at 09:10:49

But then no more:

until 09:12:11:

OK, so it’s now showing as being a couple of minutes later than previously expected.
But what happened between 09:04 and 09:12 to make the API think it should be telling everybody that this bus would not arrive at all?

That’s exactly the same as the 97 problem.

And it’s repeated with LX11BGU:

which disappears 1 minute later:

as does LX61DAA a couple of minutes later:

Then a new bus appears, but it’s not apparently coming for 27 minutes (despite this being a regular 7 minute service)
That’s still the only bus showing at 09:21:

when suddenly BGU reappears, with 0 minutes notice:

That’s not much help if you live 2 minutes from that bus stop, but were told there would be no bus for 25 minutes. It’s coming only a few seconds later than originally predicted ten minutes earler. But again, we need to find out why the API thought from 09:12 to 09:21 that it would not be coming at all.

[harry]

@harry

Doing some further tests, I find the problem isn’t only with route 97. Other buses starting from Chingford Station also have the problem.

@jamesevans

It’s now something like 9 months since I raised this problem but it appears not to have even been investigated. There is clearly something completely wrong with either the data coming from buses leaving the Chingford terminus or with what TFL is doing with that data when it presents it via the API and the countdown screens.

I initially thought it was a problem specific to 97, I’ve quoted evidence that it applies to other services, from different operators, on routes starting from Chingford. That includes at least 212, 197 and 313.

The predictions API, as currently implemented, is clearly completely unfit for its purpose, at least in respect of the first half dozen stops of buses starting from Chingford.

Could I please ask for a specific New Years Resolution, to get this unnecessary problem fully and properly investigated and fixed.