@mjcarchive yes unfortunately no change. i’ve just raised an incident for our internal teams to look at further. i suspect it’s an issue with the new system (Adiona) which exports the files but it could also be a failure with other components such as the file transfer job which runs every morning at 6am. Will post an update here once I have further news.
@neamanshafiq
Presumably no progress to report as there is still nowt. If the files exist is there any workaround for making them available?
@neamanshafi
Aren’t the tfl bus timetables also copied in to https://data.bus-data.dft.gov.uk/timetable/download/bulk_archive ?
See Sign in
@briantist
Thanks. Just 1.1Gb downloaded…
I work with the WTTs as PDFs. As far as I can see whatever is in the BODDS file is in xml format, as used in the Datastore Journey Planner zip file, which is still available from TfL (admittedly after a good kick this week). I don’t know that everything is there anyway; for Abellio I see data for 407, 464 and R68 and nothing else. If this is because they are the only Abellio routes crossing outside London then that implies that the bulk of the TfL data will not be in this source.
The JP xml files are not very easy to understand, as there can be several versions available at once for a route, the day types differ, dates of operation and non-operation require a PhD to understand and there is sometimes ambiguity about post-midnight trips. Apart from that they are fine!
Except I am having doubt about even the JP file’s completeness. For example the 164 started a new contract last weekend. Saturday and Sunday WTTs (with changed times) were uploaded before the WTT system collapsed but the only recent new file on the JP zip file is a Boxing Day special! This is not unique; I don’t think anything has emerged for the 93 either and I observed the same for another route (number escapes me) a couple of weeks ago.
1 Like
Perhaps it’s time to write a transxchange → pdf converter?
And you’re not wrong about the random nature of the date types. It took me 126 lines of compact code to decode them!
B
@briantist
The end product is Excel. I have stuff in place which converts pdf to xlsx and does comparisons with previous editions. I also have stuff which converts the xml to xlsx but for the reasons I set out it is much harder to sort out the data properly and therefore to make comparisons. An xml to pdf converter would not help unless it put everything in the right place in the resultant pdf! If I have to write anything more it would be to try and get on top of the xml-to-xlsx conversion but I don’t think I will be doing anything like that to cover what is (I hope) a short term blip in WTT availability.
As an aside, it used to be fairly easy to line up the JP Datastore files with the equivalent WTTs because the file names included the same 5 digit code (Service Change Number, I believe). Since the system changed last summer, although the Datastore files still generally contain a 5 digit code, it seems to be “random past code for this route plus 1, 2, 3 or whatever”, thus it is next to useless. I have no idea why this decoupling has occurred, or whether it matters in practical terms.
For what it’s worth the weekly Tfl API updates have been AWOL since 29/11 as well. From what @neamanshafiq said, it looks as if something rather fundamental has been trashed by a systems update
Separately, I take the XML stuff from Tfl, TNDS and BODS, as well as the data from the Tfl API, together with the National Rail API into a database. I’m working on a 10-way comparison ( 5 x Route, 5 x Timetable) to try and give a human-comprehensible picture of what differs and where. The idea is to try and spot the underlying causes of the errors and to suggest areas where investigation might be useful further uback in the generation process
@nickp
That sounds ominous if the TfL updates have stopped as well. One of the things I learnt gradually over the years was that if an error occurs at a particular point in a suite of processes, the underlying cause may lie many steps further back.
It would certainly have been useful if TfL had been able to provide an overview, comprehensive in terms of bodies of data rather than in detail, of what comes in (largely via TransXChange?) and what goes out (some via TransXChange). The working timetables are yet another output to be thrown into the mix but I remain hazy about how they fit in.
Something else that may or may not be connected.
Background first. The xml files in the JP Datastore file describe every stop as PTP or OTH. PTP means that it is used as a main timing point when only a selection of timing points is shown in a timetable, on a driver’s duty card etc. The convention is (or at least was) also used on Traveline to enable the user to toggle between all stops and main stops. The running times between stops were shown in seconds but they were constrained in such a way that the running time between each pair of PTP stops was a whole number of minutes.
In early November this changed for all newly created files. Every stop is now shown as PTP. As a consequence the running time between each pair of stops is now a whole number of minutes. This makes it much harder for a third party to identify and extract data relating just to the main timing points; it is hard to see how it could be automated without reference to an earlier list of stops for the route - and what is the use of that if the stops change, as they do?
This may not matter at all in Journey Planner terms. However the same xml files are also being supplied to Traveline. I am not sure if they or their successors offer main vs all stops options for TfL routes (Traveline used to divert users to TfL’s stop specific timetables) but if they wanted to they would surely have the same problems.
I should emphasise that the updated versions of pre-existing files are not affected; they still differentiate PTP from OTH.
So - why the change? Has someone decided that the extra complexities were unnecessary for TfL’s own needs? Is is linked with Adiona and/or the decoupling of the Service Change Number link noted in an earlier post?
I hope that you have kept the receipt for Adiona, by the way… 
It’s been around for years (and was last updated in 2014 
)
See the first link in the PDF:
http://naptan.dft.gov.uk/transxchange/publisher/2.4_6/TransXChangePublisher-2.4_6-desktop.zip
It needs Java to run.
Simon
Simon - Yes, I’ve looked at that in the past. I was thinking more of how the various TfL systems and outputs hang together and which inputs (TransXChange or other) they rely upon. In particular which inputs are common to more than output. Interdependencies and communalities, basically.
1 Like
@neamanshafiq and all
The WTT update did run this morning (Sunday) but stopped again. This time much further on, at N15.
So that should mean that day services for P4 and above are missing (including river and tube replacement services). Also night services on N2-N9, N16-N98, N155-N551 and on 24 hour routes 2-6, 23-94 and 159 upwards. So a majority rather than (as last week) minority of files are present.
The point at which it throws a wobbly seems to be random, which is not going to help in tracking the fault down.
@neamanshafiq and all
Appear to have a complete set of WTTs again this morning but unless what is happening is understood the issue is likely to recur, I fear.
Just in case it is relevant, I observe that only the files missing yesterday are dated 12th. The others are still dated 11th (in a few cases earlier) so it looks like the rerun was not recreating those files again from scratch. Might that mean that it is the file creation rather than transfer procedure which is falling over?
Thanks @mjcarchive - I’ve added this update to the ticket I’ve opened with our internal support team. I’m going to ask a separate team to check the total number of files transferred on 11th and 12th and whether the file transfer only pushes up any new/changed files.
@neamanshafiq
The file transfer usually overwrites everything - except of course when there is no new file to replace the old. It would actually make life easier if it did just push up genuinely new files, as just the new ones could be downloaded by users, but clearly the files themselves are recreated each week (often with tiny variations in size) so checking each file for newness would be an extra step.
At the moment there are 3041 WTTs listed in the data bucket.
748 of these are dated 12th
2289 are dated 11th
4 are dated 4th (they have been replaced by different day types, thus not overwritten and will no doubt be deleted shortly)
1 Like
@neamanshafiq
All I wanted for Christmas was a complete WTT update…
This week it seems to have stopped in the middle of doing N205, so similar but not identical to last week’s initial attempt.
And in my case a specific New Years Resolution to try to find and fix the serious fault reported in March at Buses that disappear from predictions API which makes the API, journey planner and countdown screens totally unfit for their purpose, at least in respect of the first half dozen stops for buses leaving the Chingford terminus.
Maybe we’ll both get them, but it will probably be for 2026 or 2028.
@neamanshafiq
The update appears to have been completed this morning.
I don’t think it is complete on the seasonals, as a lot of routes which had Ce WTTs last year do not have them yet this year, including some like route 1 where the timetable very obviously differs from Sa in the latter half of the day. If it appears next week it will be too late to be of other than historical interest! (Also see other thread.)
But that is not an issue about the update process itself.
I wasn’t expecting a Christmas Day update! It fell over after H9 this time by the looks of it as files after that are all dated 26th.
It can’t be something daft like the maximum number of files being exceeded. Can it? IIRC that was the cause of the Datastore JP files update falling over a few years back.
Seasons’ greetings, anyway.
Collapsed while doing the 77 this week.
1 Like