HI everyone,
We’ve decided to block requests with no user agent set in the request headers from the 4th September.
This is for security purposes and if a blank user agent is provided, a 403 will be returned to the requestor.
You will need to ensure your applications specify a user agent if you still require use of the API after this date.
Please let me know if you have any questions.
Thanks,
James
Hi @jamesevans I think you should allow more time particularly for devs using the unified api in their apps. Apps users are slow to update specially on Android and I can foresee bad ratings coming from users whose app suddenly stopped working. So please consider at least a couple of months or three for this kind of critical changes. Thank you.
@nakkore We would love to give a longer lead time but there is some urgency to get this change in place. I’ve contacted all the main transport apps that use our data and they are happy with the timeframe we’ve given them. Apologies to anyone else who is inconvenienced. We try our hardest to avoid changes that could break third party apps as we know our customers rely on them but in this instance we had no choice.
For those of us who use the API as back-end processes, is there any particular thing you would like as a the user-interface parameter?
I think my personal default is to use User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)
Does using file_get_contents() and/or curl in PHP generates a “User-Agent” automatically?
Hi @briantist
As long as it’s populated, you should be OK. Sometimes we block particular user agents if there’s malicious or suspicious activity associated with it, but that’s quite rare.
In terms of PHP curls, I’m not sure what’s set at default, but you can specify the user agent in your curl command
Thanks
James
Thanks. If there is no default User-Agent created by PHP, then I guess I’m going to have to contact MTR Crossrail and tell them their customer app is going to stop working…
Brian
Sorry, I know someone is going to ask… is there a staging API server that we can use to test software against this change?
I’m not sure what will happen to existing code as I’m not expecting a 403 back, so didn’t code for it.
Agree, that ought to be done. I’m assuming firefox (desktop and android) will send a user agent unless I’ve told it not to, but I’m not sure how I would test it in both cases. A test server would confirm it properly.
Thanks Brian. I found almost exactly the same using F12 in firefox.
[And yes, it did show a user agent, so I suspect that will apply to anybody else using firefox].
I’m working on setting up a test domain for this. It’s proving tricky as it’s a global firewall setting for all our outward-facing web applications. I’ll let you know how I get on.
@briantist - curl on the command line does seem to send a user agent in the headers. You can see this by using curl -v
> GET /Line/85/Route HTTP/2
> Host: api.tfl.gov.uk
> User-Agent: curl/7.54.0It looks like PHP’s built-in CURL doesn’t add an it’s own user-agent. Here’s some code…
$strURL = "https:/api.tfl.gov.uk/Line/85/Route";
$strCacheFile = "tmp.saved";
$ch = curl_init($strURL);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$fh = fopen("$strCacheFile", "w");
curl_setopt($ch, CURLOPT_FILE, $fh);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
curl_setopt($ch, CURLOPT_VERBOSE, 5);
curl_exec($ch);
fclose($fh);
curl_close($ch);
This is the output…
I note, however, that my standard CURL routine (in my “framework”) does include
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
which shows up like this…
You can specify a user agent using -A argument in curl too.
curl -A "Briantist" -v https://api.tfl.gov.uk/Line/150/Route
which produces the header:
> GET /Line/150/Route HTTP/2
> Host: api.tfl.gov.uk
> User-Agent: BriantistTrue, but I never use the command line in any of my code. PHP has always had a cURL system baked in PHP: cURL - Manual
Also, in PHP you can always use file_get_contents(“http://…”) to do any simple GET. PHP: file_get_contents - Manual