Very weird. I had it working. Now it isn’t again. I’m going to hit my head against the wall for a bit and take another look.
Thanks,
James
Sorry, not checking this every day. If you can use curl from the command line then this should do the trick:
curl --request HEAD
–url https://tfl.gov.uk/tfl/syndication/feeds/journey-planner-timetables.zip
Mark
This (in a DOS window)
curl --request HEAD -url http://www.tfl.gov.uk/tfl/syndication/feeds/journey-planner-timetables.zip
returns
Enter host password for user ‘rl’:
This
curl -I http://www.tfl.gov.uk/tfl/syndication/feeds/journey-planner-timetables.zip
generates header-type information
HTTP/1.1 302 https://tfl.gov.uk/tfl/syndication/feeds/journey-planner-timetables.zip
Date: Fri, 20 Mar 2020 00:43:52 GMT
Connection: keep-alive
Set-Cookie: __cfduid=d55b0d6972352b891dd9ad0cb8a59e32c1584665032; expires=Sun, 19-Apr-20 00:43:52 GMT; path=/; domain=.tfl.gov.uk; HttpOnly; SameSite=Lax
Accept-Ranges: bytes
Age: 0
Cache-Control: , s-maxage=
Location: https://tfl.gov.uk/tfl/syndication/feeds/journey-planner-timetables.zip
Via: 1.1 varnish
X-Banning:
X-Cache: MISS
X-Proxy-Connection: unset
X-Varnish: 1818652451
CF-Cache-Status: DYNAMIC
Server: cloudflare
CF-RAY: 576b66c48b39f3ef-LHR
but no sign of a Last Modified date.
I must be missing something here!
Ah sorry, yes - the cookie, I should have tried it, I use an app called Insomnia to make these requests rather than use curl, another called Postman is a very popular alternative:
curl -I --url https://tfl.gov.uk/tfl/syndication/feeds/journey-planner-timetables.zip --cookie __cfduid=d55b0d6972352b891dd9ad0cb8a59e32c1584665032
It seems they don’t give the actual response back until you’ve included the cookie they ask you to set in that first response. Also note that particular cookie value has an expiry date of 19th April, so occasionally you’ll need to get a new one.
Thanks, Mark. That worked, giving a Last Modified date of 19th March, though unfortunately it still contains the 10th March versions. Goodness knows whether this was the version that James said was ok, or has overwritten said version. I dare say that anything loaded now would bear little resemblance to what is expected to run next week, never mind what actually runs and it can hardly be priority now to dig further.
Michael
@mjcarchive - we’ve narrowed this down to an issue in AWS S3. A ticket has been raised with Amazon to look at this.
Thanks,
James
No file for 24th either.
A file for London dated 24th was provided to Traveline though file naming conventions and actual contents differ from what is available through Datastore. Also coverage - it seems to include rail replacement (UL…) services, river buses and files which look like they relate to diversions for football matches which are obviously not now going to happen.
I’m not sure whether Traveline actually use this as if you try and get a timetable for a TfL route from Traveline you are given a choice of stop specific timetables, with all their faults. Perhaps Traveline uses it when people are planning specific journeys?
I’ve found the cause of the issue. There is a script that’s replacing the file with an old version. I should be able to resolve this tonight, if not, I know how to force it up!
Thanks
James
We have the latest file up (24th March). Thanks for your patience on this.
James
James
Sorry to bring it back down again.
I just downloaded the file. I am afraid what I have downloaded is still the 10th. I am downloading by going to bookmark url http://www.tfl.gov.uk/tfl/syndication/feeds/journey-planner-timetables.zip, which starts the download, so I don;lt think it is anything like cacheing at my end.
My mind is boggled at the thought of a script which is designed to undo what you have just done without asking you first!
Michael
Hi @mjcarchive
Some clever person who has since left TfL was playing around with automating the upload a couple of years ago. We changed the location from where the upload was taking place due to an upgrade we’re doing. The old file (from 10th) was still in the old location and being synched, overwriting the new file. I’ve disabled the scripts permissions to upload to this folder.
I can’t recreate the file coming back from the 10th and the version that I uploaded at 7pm last night is still in our S3 storage.
If you are downloading in a browser or anything else, it does have quite a large maxage so I’d suggest purging any cache on your end and trying without the www redirect also:
https://tfl.gov.uk/tfl/syndication/feeds/journey-planner-timetables.zip
Thanks,
James
I would recommend https://www.syncovery.com/ for doing that kind of thing. It’s easy to use and works with things like S3 out of the box. And it has great log files too.
Still no joy, despite clearing the cache and removing the redirect.
Using the CURL method outlined above I get a Last Modified of 24 March at 18:20:44, not anything from 25 March.
James - I’m not sure whether you are saying that I should be getting the right file from the link, or whether I shouldn’t because it is in S3 storage (something else I know very little about).
Brian - Syncovery (something else I know very little about).- was that suggestion for me or for James?
Michael
Sorry, the Syncovery message was for @jamesevans . I’m sure he’s coded up something, but I’ve set up systems using Syncovery and they have worked reliably for years without intervention.
@mjcarchive - Can you please try again. I think I’ve fixed it now.
curl https://tfl.gov.uk/tfl/syndication/feeds/journey-planner-timetables.zip --output JP.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 133M 0 133M 0 0 45.4M 0 --:–:-- 0:00:02 --:–:-- 45.4M
New file is 133MB (old is 138MB)
Success!
Thanks for your perseverance. I dare say that many of the timetables will be overtaken by events pretty quickly but when/if things return to predictability, even normality, let’s hope this particular problem has been seen off.
Michael
No problem. I’ve learnt a lot about how this has been set up in the last few days. I’ll be re-engineering it in the near future.
J
Just the WTTs and the spider maps to re-engineer then, James!
To be fair, the WTT upload is (or was until this week) working without creating new errors, so someone just needs the correct the residual errors, if we ever get back to normality.
Michael