Missing Northern southbound data on Sunday 2020-03-01 after 23:10?

Unified API
1

Hi,

we are consumers of the TfL Tube Arrivals service. We are using the TfL Unified API, primarily the Streaming (SignalR API) version with the Polling (HTTP polling API) version as a backup.

Based on a user report, it appears that there are missing arrival predictions for Northern line trains, in particular departing from High Barnet, after approximately 23:10 a few days ago on Sunday 2020-03-01.

Specifically, we have recorded these two train departures from High Barnet:

  • Train 117 departed HB at approximately 23:05, headsign Morden via CX,
  • Train 137 departed HB at approximately 23:09, headsign Morden via Bank

However, after triple-checking both our application logs, as well as the incoming raw SignalR Stream feed messages, we have been unable to find any further departures (other than the above two trains), or other southbound train movements, for Northern line trains from High Barnet.

Additionally, we have been unable to see any train departures after 23:11, or other southbound train movements, for Northern line trains, from Edgware other than the below trains:

  • Train 016 departed Edgware at approximately 23:03, headsign Morden via CX,
  • Train 045 departed Edgware at approximately 23:07, headsign Morden via Bank,
  • Train 046 departed Edgware at approximately 23:11, headsign Morden via CX

On the other hand, northbound trains (e.g. from Morden, from Kennington, etc.) seemed to have been tracked fine, including departures and ongoing trains.

Note that when I say “train departure”, due to how the Arrivals feed works, I simply mean the first sent message for that train’s just-started trip; this can either be sent as the train is leaving the station, or as it’s already on the way to the second, or even the third station; we have found no messages of any kind for these trains which presumably were running in reality.

Our user reported that they have been on a train from High Barnet to East Finchley, that departed High Barnet at approximately 23:30, which is in line with this scheduled trip:

Similarly, we can see that the only scheduled trains going to for example Kennington are trains that depart at 23:04 and 23:08, corresponding to the two trains we have tracked starts of mentioned above (trains 117 and 137).

Can anyone correlate the above findings? If so, is this perhaps a bug in the Tube Arrivals system that maybe doesn’t send predictions for trains going from High Barnet to East Finchley, or something like that?

We have noticed that at the time in question, Citymapper had real time train predictions for numerous trains going in that direction, e.g. at 23:39 from Finchley Central to East Finchley there were train departures “Due”, “Due” (apparent duplicate?), “5 min”, “9 min”, and “23:55”; other than the 23:55 time, the other times were marked with the “real time” icon, suggesting they were tracked by a real time feed.

Is there perhaps a third feed that we are unaware of, other than the above mentioned SignalR API feed and HTTP polling API?

Thanks!

2

Hi

The problem that we all have with terminal stations is that that the data we really want is Departures, but the system in the Unified API is Arrivals.

Have you tried cross-referencing the data with Tackernet? The output is XML

http://cloud.tfl.gov.uk/TrackerNet/PredictionDetailed/N/HBT

3

Hi,

that’s fine, our system is handling the predictions as Arrivals.
Yes, it would be nice to have Departures too, like in Darwin.

I’m not sure what you are referring to, though: if you mean that a fundamental issue of the system is that it only starts sending predictions for a train once it starts moving, our application accounts for that, as presumably all feed consumers have to as well.

The issue in question is that there were no predictions of any kind for any trains that have departed HB or Edgware after 23:10, e.g. for that train I linked the static timetable schedule for, a train departing from HB at 23:30;
while the TfL Tube Arrivals predictions system wouldn’t send any predictions until the train leaves HB, normally we would start getting them at let’s say 23:31 or so.
However the issue is that there were no such predictions of any kind for the trains in question; only predictions for trains that have departed from HB already prior to 23:10
(the trains previously listed).

Regarding the link you provided,
we are currently using the following data sources:

We are also using a TrackerNet feed, however not for Arrivals, but for line statuses only (http://cloud.tfl.gov.uk/TrackerNet/LineStatus/IncidentsOnly). I’m not sure what is the connection between the feed you provided and the above mentioned feeds (i.e. which data sources do each of them use?) - thanks for the link though, we will analyze it. Do you know if there is any historical data snapshot repository for it (e.g. a snapshot of the data every 5-10-15 minutes), so we can check whether it contained the missing predictions in question
(in other words - can the feed you linked contain more/different data than the above mentioned two data sources, which in our experience seem to be based on the same “common” data source)?

I wonder why the TrackerNet “PredictionDetailed” feed linked above doesn’t seem to be linked at https://api-portal.tfl.gov.uk/docs, and is there maybe any “central feed repository” for TfL feeds that contains all types of arrival prediction feeds (like in e.g. Darwin which is well documented)?

Thanks!

4

Sorry, I don’t mean that the issue is with the dwell time (ata to atd time in Darwin speak). Most of the TfL API systems don’t know about departures from terminals until they have happened, or are a few seconds away from happening.

You can see in the system any number of trains that are on their way to a destination! However you can’t see what those trains are going to do once they get there. There are on-platform and in-station indicators, but that information doesn’t make it into the API data, either the Unified one or TrackerNet.

The issue is that there isn’t anything public showing what the planned train turnarounds are, that train 111 will become train 222 and depart at a specific time.

To be fair, you don’t actually know that a train will depart in Darwin until it has a platform number announced, and this is usually 10 minutes before departure time.

The problem is that most of the tube isn’t running quite to time, and all passengers really care about is the headway times as they just want to turn-up-and-go.

Anyway, it is annoying because there are terminal stations in Zone 1 (Edgeware Road D&C) Zone 2 (Stratford, Brixton, Hammersmith H&C) with high levels of trains and you can’t show this data to users.

5

The Trackernet guide is here - http://content.tfl.gov.uk/trackernet-data-services-guide-beta.pdf

1 Like
6

Thank you, we will check it out.

I wonder if there any documentation where the two types of feeds get their data from (the Unified API we are consuming and this TrackerNet Arrivals feed)?
In other words, is there a chance this TrackerNeed feed contains data (real time train movements) that might be missing in the Unified API?

Hopefully someone also knows something about the original question regarding the missing southbound Northern line train movement data in the Unified API feed.

Thanks!

1 Like
7

I think someone has probably explained it on this forum or the Tfl Data blog - https://blog.tfl.gov.uk/

Yes, but also note the caviate

<Disclaimer>This system is an INFORMATION ONLY system, relying entirely on information
 received from the relevant Operational Railway Control System(s). It is NOT considered a 
safety related system in the Railway Engineering sense. However, the information reported
may trigger user intervention by staff regarding possible incidents on the railway, and 
confirmation of the situation observed should be obtained prior to any corrective action being 
taken.</Disclaimer>

@jamesevans will no doubt look at it when he’s in the office.

1 Like
8

Small update:

We have noticed that Northern line trips from High Barnet to East Finchley also happen on other days of the week.

For example,
on Monday through Thursday there is a 23:25 trip from HB to EF, as seen on the TFL timetables page:

We have checked our logs for Tuesday 2020-03-03 and can confirm we have received prediction messages for the 23:23 trip from HB to Morden, as well as the 23:27 trip from HB to Kennington, but none for the 23:25 trip from HB to East Finchley.

We have also done a broad search in our logs and couldn’t find any trips going to East Finchley during the whole day.

So perhaps we can assume that predictions messages for vehicle on any trip from HB to East Finchley are not sent via the feed?

I will make sure to try out the TrackerNet PredictionDetailed feed to see if whether it contains the missing data.

9

@Shortcut

I’m guessing that this is all related to the Northern Line “Highgate” sidings being just south of East Finchley, so any HB to EF journey will be a train going into the sidings and so filtered out from public consumption?

image

10

Perhaps, but these are still regular, “real” trips that can be taken by passengers, that should appear in the real-time feeds, right?

11

The operational issue is a question of staffing (or the driver leaving the train) on Platform 3 at East Finchley as this platform is the exclusive access to the sidings, or is it operational easier to use the staff on 1/2/3/4 at High Barnet to clear the train of passengers?

When I look at East Finchley Underground Station - Transport for London I can’t see any trains that are shown as arriving at platforms other than 1 and 4.

From memory, there are on-station staff facilities at HB.

(It’s not a problem from EF Plat 2 as it’s an empty train coming into service).

12

@Shortcut

Looking at the Working Timetable the trains do show as being in service

image

However, there is not much movement to/from the Highgate Depot outside first and last.

image
image363Ă—542 25.4 KB

13

The user report included this picture of trains departing High Barnet on 2020-03-01 (Sunday):

User_Report_HighBarnet
User_Report_HighBarnet750Ă—766 141 KB

14

That matches up with the (yeah, last page!) WTT for Sunday.

image
image1148Ă—146 35 KB

image
image1147Ă—134 31.1 KB

image

1 Like
15

@Shortcut

This, by the way, might just be back to the question of “do the terminal stations display boards just show the timetable?” rather than live departure data.

16

We can see that the TrackerNet feed has trains departing HB to East Finchley - for example:

scheduled trip departing HB at 23:14 going to East Finchley:

Latest data right now in the TrackerNet PredictionsDetailed feed as of 23:14:

http://cloud.tfl.gov.uk/TrackerNet/PredictionDetailed/N/EFY

Extract of the data:

<T TrainId="14787002" LCID="205" SetNo="154" TripNo="15" SecondsTo="682" TimeTo="11:00" Location="Between High Barnet and Totteridge &amp; Whetstone" Destination="East Finchley" InputDest="EFY3 (EFY3//EFY3) [=EFN3/EFY3=A]" DestCode="791" Order="0" DepartTime="23:14:45" DepartInterval="682" Departed="0" Direction="0" IsStalled="0" TrackCode="TN50365" LN="N" LeadingCarNo="51609" />

I cannot find the same train entry in the Unified API feed.

Therefore this is clearly a bug in the Unified API feed.

1 Like
17

Agreed. Let’s see what @jamesevans has to say tomorrow.

1 Like