Hi @jamesevans
The 3-second one finished about 30 mins ago, the 1-second one was this morning. The one for the update dated 4/5/2020, without the tick delay but with the 3-second pause for retries, was in the small hours of the morning of the 5th. I can run it again with a time stamp in the log if you want.
Some observations based on the results of the update dated 21/7/202, following the changes at the Tfl end.
Code 400 errors still occurring, as originally described.
The number of 500 errors is significantly reduced, down to 288.
âDeparture stop does not match originatorâ discrepancies still occurring, as before.
Still the same âstop has been removedâ errors, but I note that this includes route 521 in both directions. This was added manually to the Tfl build and may thus give a pointer to why/how these errors arise.
No change on the rail front, but my comment on 521 may be relevant here as well.
I think this might be related, and apologies if this has already been mentioned. But Iâve seen random 500 errors returned also for the timetable endpoint:
Endpoint: https://api.tfl.gov.uk/Line/36/Timetable/490010204K
Response:
{
â$typeâ: âTfl.Api.Presentation.Entities.ApiError, Tfl.Api.Presentation.Entitiesâ,
âtimestampUtcâ: â2020-07-27T22:50:59.461446Zâ,
âexceptionTypeâ: âApiExceptionâ,
âhttpStatusCodeâ: 500,
âhttpStatusâ: âInternalServerErrorâ,
ârelativeUriâ: â/Line/36/Timetable/490010204K?app_id=2dc44de8&app_key=dc8ca8f52b3ab183573454d7d30fa83câ,
âmessageâ: âNo timetable available for 490010204K > 1010204, inbound > R, 36 > 45036.â
}
Itâd say about 80% of the time it returns a 200 code just fine, but other times there are internal errors.
Result of update dated 8/9/2020:
All the usual suspects still in place, but we have a new one (everything after âHTTP 400:â is the returned response):
Error loading timetable data for 404/490002178Z/outbound: ##HTTP 400: âBad Requestâ Response: â{"$type":âTfl.Api.Presentation.Entities.ApiError, Tfl.Api.Presentation.Entitiesâ,âtimestampUtcâ:â2020-09-09T13:42:58.1556113Zâ,âexceptionTypeâ:âApiArgumentExceptionâ,âhttpStatusCodeâ:400,âhttpStatusâ:âBadRequestâ,ârelativeUriâ:"/Line/404/Timetable/490002178Z?direction=outbound&app_id=xxxxxxxx&app_key=yyyyyyyyyyyyyyyyyyyyyyyyy",âmessageâ:"Could not find a matching ICS code for the Naptan Id 490002178Z. Argument name: fromNaptanId. "}â
I note that 4900029178Z is part of a contiguous block of StopPoints from 490002164HN to 490002189Z that are all missing ICS codes
Iâm also getting similar transient 500s for /Timetable. Happy to help if you want to request anything of me. For now, we will use a similar brute force approach to above and reduce dependency using local filesystem as we only need timetables for durations between stops.
Just to confirm what @lbennett has said, there seems currently to be about a 1-in-3 chance of getting an error (mostly 500s, with a handful of 503s and time-outs). As an example the run following the 24/11/2020 update produced the results shown below.
Note that all other errors reported earlier are still outstanding.
Out of a total of 1914 requests, the numbers of errors in successive passes was;
| Pass | Errors |
| 1 | 643 |
| 2 | 268 |
| 3 | 123 |
| 4 | 55 |
| 5 | 26 |
| 6 | 10 |
| 7 | 3 |
| 8 | 2 |
| 9 | 1 |
| 10 | 1 |
| 11 | 1 |
| 12 | 0 |
@neamanshafiq has raised an investigation with our Journey Planner vendor to look into the 5xx messages weâre getting back intermittently from the timetable API.
Thanks,
James
Well, I donât know what theyâve done, but itâs definitely not an improvement.
Errors per pass following the 15/12/2020 update (out of an initial total of 1895):
1641/771/238/81/33/10/2/2/0
With a 1st pass failure rate of 87%, I would question your use of the word âintermittentlyâ
It was also extremely slow - running with a 10-second timeout, the 1st pass took from 20:38 to 23:07 last night, with the 8th pass complete at 00:37 this morning (9th pass waited on user response, so delayed until 01:04). This compares with 20:00-20:33 (2 timeouts and 2 503s in the first pass) for the equivalent set of requests on Line/Route/Sequence.
@jamesevans: Following further investigation, hereâs an update on what I believe to be the outstanding issues with /Line/Timetable.
All data based on the update dated 25/5/2021.
All data - undocumented parameter
/Line/Timetable is documented as being /Line/{lineid}/Timetable/{fromStopPoint}, but it acually requires an additional ?direction={direction}
All data - 500 errors
A significant proportion of the requests return an error 500 âInternal Server Errorâ. This can be overcome by brute-force retry. For the date concerned, 647 out of 1907 requests returned an error on the first pass, nearly all 500s, with a scattering of 503 and timeout errors. Subsequent passes reduced the number of errors as follows:
647, 323, 151, 62, 34, 11, 6, 6, 4, 3, 1, 1, 1, 1, 1, 1, 0
In case itâs relevant, the request with an error in every pass was 286/490011319HB/outbound
Nonâexistent stop Ids - 400 errors
An orignating stop in the list returned by /Line/Route is rejected by /Line/Timetable with error code 400 and message âCould not find a matching ICS code for the Naptan Id xxxxxxxxxxx. Argument name: fromNaptanId.â
The only current example of this is cross-country/0000BRSTLTM1/inbound, but I note that there are 63 such stops, all of them rail, in the /Line/Route/Sequence data
Rail - No route sections
All the rail-related line/route/direction entries returned by /Line/Route are returned by /Line/Timetable with no route sections and the error message âThe stop you selected has now been removed from the route, and therefore we cannot show you a timetable. The route page will be updated shortly to reflect these changes.â
It should be noted that the stations and stops sections of the returned data appear to be correctly populated
Bus - no route sections
The same problem arises with a handful of bus entries, in this case only two entries:
412/490013142C8/inbound and 434/40004410113A/inbound. If you can spot what is different about these two, it may give you a handle on the rail issue
Bus - no interval sets
I have only seen this with hereeast-shuttle/490001338Z/outbound. I note that this is a short, circular, outbound-only route and I suspect that this is related to the more genreal problem with looping or divergent routes.
Bus - departure stop does not match originator
This is where the stop returned by /Line/Route and used as a parameter to /Line/Timetable is not the same as the departureStop in the timetable section of the returned data. In the cases that Iâve examined, the correct sequence appears to be originator/departureStop/remaining stops in the interval set. The first three instances of this are:
18/490000077E/inbound - departure stop 490000077H
24/490015832E/outbound - departure stop 490011003E
40/490006232W1/outbound - departure stop 490015428N
This may turn out to be a special case of the more general missing stop problem below.
Bus - missing 2nd stop
There are over 300 instances where the second stop in sequence is missing. This is the first stop in the interval set, after originator/departureStop. This affects lines 2/inbound, 3/both/, 4/outbound, 6/outbound, 7/both and many more. Again this may be a special case of the more general missing stop problem.
Bus - general missing stop problem
There are just under 200 other instances of both stops other than the 2nd and multiple stopns missing from the timetable. Examples are 27/outbound, 81/both, 96/outbound, 101/outbound, 107/both.
This, and the previous two categories, could all be related to the issue of stops in the same StopArea that you have mentioned previously.
Bus - duplicate stop
There are a number of instances where the same stop occurs twice in succession in an interval set. These are:
490000054CA in 125/outbound/0
40004402075A in 203/inbound/0
490011630S2 in 366/outbound/0
490011630S2 in 366/outbound/1
490002115Z in 483/inbound/0
930GTMP in city-cruises/outbound/0 - I know this isnât a bus
Bus - multiple routes with missing stops
This may be a variant of the issues above. At least two routes are present with stops missing, typically in one of them. Unlike the TxC files, there are no start and end dates so I can only assume that all routes apply to the current period.
Examples are:
18/inbound, 40/outbound, 125/inbound, 150/outbound
Bus - miscellaneous
20/inbound: tt is wrong at Loughton, also missing 2nd stop
Route: âŚ/1500IM504/150042015006/1500SAINS1/1500LOUGHTON /150042010002/âŚ
Timetbl: âŚ/1500IM504/150042015006/1500SAINS1/ 1500IM504/150042010002/âŚ
20/outbound: tt is also wrong at Loughton
Route: âŚ/1500IM504/150042015007/1500SAINS1/150042010004/ 150042010005/âŚ
Timetbl:âŚ/1500IM504/150042015007/1500SAINS1/ 1500IM504/150042010005/âŚ
108/outbound: tt missing 2nd entry, tt & route both showing route via Tower Bridge, rather than via Blackwall Tunnel, tt also has erroneous non-stop route between Chrisp Street Market & North Greenwich Station
192/outbound: route has 490001024Z, tt has 490001024X at Bush Hill Park Station
224/outbound: tt missing 2nd stop, route has incorrect segment linkages, caused by the loop at Belmarsh Prison, should be straightforward 1>2>3
358/inbound: route has a variant path missing, tt is correct
371/outbound: tt has variants with erroneously missing stops
384/inbound: multiple differences between route & timetable
397/both: same mix-up at Loughton as line 20
401/both: route is missing a variant, tt is correct
463/outbound: route & tt differ
605/outbound: tt truncated
663/both: route & tt differ
655/both: tt is showing only a limited-stop service, route has full sequence
681/inbound: tt has limited-stop service
718/outbound: route & tt differ
B12/outbound: tt has limited-stop service
EL2/inbound: tt has limited-stop service
R5/inbound: tt has 2 variants, one missing 2nd stop, the other missing stops 2-41
R6/inbound: route has loop problem, similar to 224, tt correct
R10/inbound: tt has two variants, one missing a couple of stops, the other missing a lot
I think that R5 and R10 are mixed up with each other, as they are essentially the same route in opposite directions
DLR
Timetable is missing Poplar>Stratford via Pudding Mill Lane
River
RB1/inbound: route missing limited-stop services
RB1/outbound: route missing CAW-GNW non-stop, both missing EMB-WMR-WMP-BFR
RB2/inbound: route missing MBK-BSP non-stop
RB6/inbound: both missing GNW, route has WMR - not in timetable, both missing BSP-CHP non-stop
RB6/outbound: both missing MIL-WRF-WAS
Tube
Central/outbound: tt missing GGH-HLT
Circle/inbound: mix-up between PAC and PAH at Paddington - tt has a section PAC-RYO which is not possible
District/inbound: tt missing ECT-KOY
Metropolitan/both: route missing fasts and semi-fasts
Metropolitan/outbound: route has Willesden Green stop, missing from tt
Piccadilly/both: route missing non-stop TNG
Iâve been using this myself in the last week and it does seem that, despite having a 500-requests-a-minute limit to my calls for each of my projects, I quickly start seeing the error message saying Iâm over my rate limit if I donât wait 5 seconds after each call.
Something appears to have changed on or about 17:00 on 18/11/2021.
Background info:
I download all the timetables through the API in numerical/alphabetical order. Any 500/503/530/timeout errors are stacked up and processed in a subsequent pass. This is repeated until the download is complete.
During pass 1, there were only timeout errors until line 679, only 503 errors until merseyrail, and then only timeout errors to the end.
Subsequent passes had only timeout errors with a much better drop off rate than heretofore - number of errors per pass was 702/149/26/2/1/0
The first pass took from 16:32 to 18:02, due to all the timeouts, so any changes made during that period may be pointers to the underlying issue.
Actually it changed about 14:00 yesterday!
After a deep dive, we found some configuration issues in our loadbalancer which we have now rectified and it looks a lot better in terms of timetable performance.
Weâre still looking at some errors on timetables, but weâre a lot happier than we were.
Thanks,
James
Itâs been one of those weeks for you, JamesâŚ
As itâs a new year, I thought it would be a good opportunity to resurrect this thread.
A lot of earlier issues are still outstanding and I shall detail them in a future post, but one problem is apparent on a very large number of lines and is swamping my error reporting, so that it is difficult to spot other errors.
In many cases, one or more stops immediately after the initial stop are missing. I suspect that these are ones with a timeToArrival of 0.
The first dozen or so occurrences are 1/outbound, 2/both, 3/inbound, 4/both, 5/outbound, 6/outbound, 11/outbound, 12/inbound, 14/outbound, 16/both
I attach a screenshot of 3/inbound as it is a good example of two consecutive stops missing
Note that this problem only occurs in /Line/Timetable - /Line/Route/Sequence has the correct data, as does the Tfl TxC file, TNDS and BODS
@nickp
Looking at the 3 TxC file, while it always.takes zero minutes to get from CP Bus Station to Parade, on most journeys it takes 1 minute to get from Parade to College Road (on some it is indeed zero). Perhaps even one zero time journey is enough to trigger something.
If this issue is related to zero time from the first stop, it is worth noting that it was only after the change to the new system (Adiona?) that this should have become a problem because before that times from the first stop were always quoted to the nearest second. Of course that is spurious accuracy but I donât know why it was changed or indeed whether it was deliberate.
I have noted whole chunks of stops mid-route on some routes where the travel time from the first stops are identical, then all of a sudden it jumps several minutes to the next stop. I donât think that has ever been explained.
Thanks for raising this issue.
Looking at the Unified APIâs code, the timetable service seems to only return stops where the elapsed time of the journey (since the stop specified in the request URL) is >0. I guess the idea is that it shouldnât return the specified stop â only subsequent stops. But that breaks down when you potentially have a 0 journey time between stops. Maybe we should just filter out the first stop rather than look at the time.
Weâll need to take this away and investigate further.
@LeonByford
Just go back to using seconds rather than whole minutes. Then no times between stops should be zero.
Just to note that, for your reference, Iâve just raised the Unified API issue as ticket #106996.
Thanks, @LeonByford , I await the happy result
@mjcarchive : I think your clumps of stops issue is probably because most routes only have certain stops as timing points, the intermediates are arbitrary, so you can be lazy and fill in the data as 0,4,4,4,4,4,6 where 0 and 6 are the timing points without being technically incorrect. Not very helpful for predictions, mind: you either have superbus or mudbus
@nickp
What you get in a few cases is more like 0,0,0,0,0,6. If this really is laziness, or lack of understanding of the system, you have to ask who is monitoring what is done.
Youâre right about timing points of course. The formal ones used in WTTs used to be distinguished in the TxC files by use of the abbreviation PTP and the running time between PTPs was always a whole number of minutes. The bulk of the stops were shown as OTH and running times for them were usually in minutes and seconds.
Note that I say âused to beâ. On newly created files since just after Adiona came in, ALL stops are described as PTPs, even though they self-evidently are not. Maybe this forces integer values, in which case the answer is to go back to PTP/OTH.
I have no idea why these changes were made. The software wasnât designed by Fujitsu, I trust!
