At present my program looks at the first two route sequence sections in the XML then stops. That suffices for all the XMLs I initially inspected and also applies to many other routes. I simply gave them the XML labels (Inbound and Outbound) and forgot that some routes have (many) more. Before modifying the program I need to find time to work out what if anything to do with the rest.
A good place to start is the two TfL bus sequence CSV downloads (regular and FOI) because my only interest in the XMLs is to produce regular updates for a similar database. On inspection they include many routes with multiple sequences:
The regular CSV names them “Runs” and numbers them all 1, 2, 3, etc with no indication of direction.
The FoI CSV names them as " Patterns" and numbers them OUT0nn and Back0nn where nn can vary from 01 to 29.
In many cases, and always in the high numbers, there is only one RouteLink so it seems safe to assume higher number ones are useless for my purposes. However, some routes do genuinely have >2 sequences. I will probably change to using OUT0nn and Back0nn instead of Outbound and Inbound. It is easy to ignore a sequence which holds < x links but what is the “correct” value of x for an accurate (useful) database?
This approach will not solve the route 57 problem with its sequence of 21421 RouteLinks! The FoI CSV has one out and four back sequences for 57, all with a sensible number of stops. I will just ignore the giant sequence. There is a second 57 xml which is much better but still has a sequence with too many stops
@misar
Taking a step back, in what circumstances should a route have more than two RouteSections? A quick look at routes like 102, 601 and SL7 suggests that any scheduled short workings get their own RouteSections but these days short workings are very much the exception, so they should not generate more than two or three extra RouteSections. The 401 (which has an extra stop in the middle on Mondays to Fridays) also gets extras. Files like the 57 obviously have far too many and I suppose how that occurs in the first place is something for TfL’s technical people to mull over, if it has any practical impact for them (it may have none, unlike the NATS data input!).
A couple of observations. The last character of the route link code may not always refer to the same pair of stops. For example on the 601, link #1 in each section is different because they start from different points on the route. In the Journey Pattern data the stops themselves are given sequence numbers and these are consistent …
… except where they are not! As you might see form my postings, very occasionally the sequence changes but the change is only made for some days of the week, so there is internal inconsistency; indeed some of the sequence numbers are then left blank. I suspect that these errors may affect Journey Planner but they are rare and usually quickly corrected once identified.
@misar@jamesevans
In connection with this, I have carried on generating my own file (not quite there yet) but I thought it might also be useful to append SMS code to a deduped list of stops. These have come from previous TfL releases plus interrogation of stop points, e g https://api.tfl.gov.uk/Stoppoint/490002118Y
I had one last go at gap filling. There aren’t many and the majority are just alighting points, so SMS codes aren’t needed. However, I did find some other stops where the call worked but no SMS code was visible. In some cases a working code could be gleaned from elsewhere but for the first group of five I could not find anything.
GSM refers to Google Maps (the SMS code is typically visible after clicking on a stop), LVF to London Vehicle Finder (which is built around SMS codes).
Five stops where GSM shows no code but does show arrivals; LVF appears not to show arrivals.
The first four are on route 339:-
490002118Y Bobby Moore Academy Primary School
490002118Z Bobby Moore Academy Primary School
490002119Y Sidings Street
490002119Z Sidings Street
The fifth is on route 303
490000427Z Colindale Superstores
Three out-county stops where GSM shows an 8 character (county) stop code. LVF has codes which work.
150042002100 - 58017 (Epping Forest College)
150042005009 - 74905 (Debden Estate)
40004406038B - 98502 (Epsom)
Two stops on the loop at the northern end of the 147. GSM shows no code. LVF has codes which work
490010870N - 12588 (Bergholt Avenue)
490015719N - 72801 (Falmouth Gardens)
One northbound stop (490005382N Clayton Road) on 63/N63 in Peckham which I am not sure exists! No sign of it on GSM and LVF does not appear to show arrivals.
One southbound stop (490012173Z Jail Lane/Single Street, SMS code on LVF is 12173) on the R8. But there is another stop 490012172S (Berry’s Green) in the stop sequence which has virtually identical coordinates, with SMS code 76207. Clearly duplicates so one should not exist!
One stop (490007057T) on the N11 at Fulham Broadway. This is the last stop for 11 and I think the N11 uses a different stop (H). If T is not a boarding point it doesn’t need a SMS code.
On the subject of set-down only stops, just because you can’t board there doesn’t mean you might not want to know when the bus is going to arrive there - you might be on the bus or waiting at the stop for someone who is on the bus and want to know when it’ll arrive…
The stops you mention for the 147 are in fact for the 366 and are virtual bus stops as they’re Hail & Ride “stops” (I think you meant 72301 as 72801 is in Battersea). As the H&R ‘stop’ could be quite long, how do you show accurate arrival times when depending on the virtual stop length it could take the bus a few minutes to traverse it? (What is the longest TfL H&R ‘stop’ section?)
The LVF 1xxxx codes are internal codes we created when the StopCode2 ATOC field didn’t have an associated value in StopCode1 SMS field - I think after the “QR code text in the SMS field” issues a while back, we imported the out of date busstop.csv file which caused us a few problems . We’ll be correcting these, where possible, as & when time permits.
The reason we’re not showing the four stops on route 339 you mention is simple, they’re not presented in the Countdown API for either route 339 as a whole or by those StopCode2 ATCO values. The Countdown API spits a dummy (throws a JBOSS error) when you try to query any of those four ATCO codes but doesn’t for the route 303 “490000427Z Colindale Superstores” example. Again, this stop isn’t mentioned at all in the Countdown API if you query the arrival times for route 303 as a whole.
The Unified API doesn’t return any arrival results for any of those five ATCO codes (al least not through APIs: Details - Transport for London - API). These might be one for @LeonByford to pick up?
490005382N - There’s a dedicated crew change over stop on Clayton Road. I suspected that since Abellio took over route 63 that it wouldn’t be used but it seems they still are as per Google StreetView. The Countdown API ‘StopPointType’ gives it a value of “STNA” and is the only stop to have this value (It’s also not listed in the Countdown API documentation)
I’m going to guess the 192 somewhere in Edmonton 192 bus route - Transport for London I understand, however, “Buses are not serving the Hail and Ride section on Plevna Road.”
I got bored and queried the complete list of stops from the Countdown API for the stop type (“stopalso=true” to return stops with no predictions), I then cross referenced that with the Countdown API documentation to get the stop type definition and whether that stop type should be displayed to the public or not.
The Yellow & Blue/grey highlighted types are stop types defined in the API documentation. Yellow exists in the Countdown API output but blue does not. The un-highlighted types are present in the Countdown API output but are not defined in the Countdown API documentation.
Curiously, of the 61 ‘SHCP’ “H&R Comm Point Start” stops which aren’t meant to be displayed to the public, 50 have SMS codes but only 47 of those 50 have ATCO codes. Eight of the stops are named “Hail & Ride Section Ends Here”.
Of the 2342 ‘null’ stops, none have SMS codes but 14 have ATCO codes.
For comparison, here’s the breakdown of the recent FoI stops list:
I am aware that some “second stops” on a route are designated as special timing points for QSI purposes because there are difficulties getting the times for the first (departure) stop. Maybe STHQ stops are dummy stops for a similar purpose (tyhe second stop being too far away?).
Yes, there would be some point in showing arrival times at last stops and quite likely that usually happens. There is more need for it at earlier stops though and that was what I wanted to focus on.
Yes, 366 (where did I get 147 from as it used to be 145?). I understand the difficulty with lengthy H&R sections but many H&R points (not the same thing, perhaps) do have SMS codes and do generate predictions, I think.
Agreed Jail Lane is a virtual bus stop but it it is located pretty much bang on the non-virtual bus shelter at Berry’s Green. Only one of the two stops should be in the southbound sequence.
I think a bus was in the way when I looked at Clayton Road on GSM! I can see it now. As it is the Datastore JP files I wonder if it could pop up on JP itself?
I was unaware of the role of the 1xxxx codes on LVF. Not sure i can get my head round how you get the data out with a dummy code!
Hi, just to let you know that I’m taking a look at these five stops.
The Colindale Superstores stop is an interesting one: marked as a “Dead Bus Stand”, and Google Street View shows a sign saying “do not stand buses here” (in March this year). But in Adiona it appears in the stop sequence for Route 303. I’ll probably need to check with colleagues who are more familiar with whether Route 303 is actually supposed to stop there or not!
With regards to those Bobby Moore Academy Primary School and Sidings Street stops, they are present within the Journey Planner timetable but not in Adiona/iBus/Countdown. A quick look at Google Maps suggests these might only be temporary stops and therefore not captured in our asset management system. I’ll try to get more answers next week.
As for the Colindale Superstores stop, although this is present in the current Journey Planner timetable, it is not present in the timetable that comes into effect tomorrow. I suspect the discrepancies between systems will resolve themselves within the next couple of weeks.
I’ve ridden the 339 around the.stadium a couple of times recently from Shadwell to Stratford and the on bus iBUS screens just show * until they reach the Aquatics centre stop.
@LeonByford
While you are here may I ask you a question about the bus stop and sequence CSV databases?
I assume they are prepared as reports generated from the main Journey Planner database. Is there any reason why updated downloads cannot be made available on a reasonably regular basis? That is particularly important for the sequences which change more frequently than the stops.
Hi, I’m personally not very familiar with these CSV files or how they’re produced.
The “our open data” web page states that the data comes from iBus, which uses a very different dataset to Journey Planner.
May I ask if there is anything in these CSV files that you need that isn’t in our TransXChange data / Unified API / NaPTAN? Or do you want the CSV files because they’re more convenient to consume?
I can’t speak for Misar but for us (LVF), it’s just very handy to have a list of a) all the bus stops and b) the bus route stop sequences in a very simple format.
One thing that would be handy is to know when bus stops are removed / moved / re-named. I believe we recently had a case whereby one of the SMS stop codes for the ImberBus route 23A which had been loaded into iBUS many years ago was re-used for a stop on TfL route 384. As we already had that stop code in our system, the system chose to use the name that it already had… (Oops), so having a definitive list of active / retired stops would have prevented this from happening as it would have been purged.
Apologies, I find it difficult to identify what lives where in your rather complex system.
Re your question, many years ago I started a program to obtain and display live bus arrivals from the API using the stop NaPTAN ATCO codes. At that time I discovered the two CSV downloads somewhere on your website. They were ideal to allow rapid searches by stop or route for the required ATCO whilst using the Internet (then limited speed and relatively high data charges) only to retrieve the arrivals at the selected stop. I augmented the TfL stop file with additional geo data (eg postcodes) for the stops to broaden the search options. However, the key benefit of the TfL stop file was (I assumed) to provide a list of every bus stop served by TfL and no others. Likewise, the sequence file with every TfL bus route.
Amazingly, using local databases is still very efficient and those files sufficed for most of my usage ever since. However, I realised recently that they are now deviating too far from reality, especially the sequences where you have been making many changes. NaPTAN’s stop downloads are updated frequently and I use them for additional useful information. Unfortunately, “490” only covers Greater London whilst “All London” extends far beyond your services (almost 3x as many stops).
Hence, I find myself in this thread discussing how to generate new CSVs from the weekly XML timetables. A complete list of unique stops should be relatively straightforward but I started with sequences as that is more pressing. I now know that extracting only “correct” sequences is far from straightforward. It was very convenient having TfL do it for me!
I’m not sure whether it’s possible to get these CSV files updated again, but thank you for your feedback; we can keep this in mind as we try to improve our provision of open data.
Extraction of a bus stop CSV list from the XML timetables was straightforward. I just downloaded the current XML zip to check that I picked up the changes at Cromwell Road BS last Monday. However, the part zips are still dated 22082023 with the files showing CreationDateTime=“2023-08-21T16:45:59.2876141+01:00” ModificationDateTime=“2023-08-21T16:45:59.2876141+01:00”.
I was under the impression from my reading here that the files are working timetables updated weekly in advance. Is that not the case?
@misar
That is odd as the zip file that I downloaded last Tuesday evening definitely had part files dated 29th. @LeonByford explained a technical issue on caching (at the TfL end, not the user’s) a few weeks back but from my understanding of his explanation that would not have any impact several days after a new set of files was created.
@mjcarchive
Thanks. I just repeated the download and now also have the zips dated 29th.
Possibly my PC failed to download the first time and I opened the old zip.
