Blank User Agents to be blocked on Unified API

HI everyone,

We’ve decided to block requests with no user agent set in the request headers from the 4th September.

This is for security purposes and if a blank user agent is provided, a 403 will be returned to the requestor.

You will need to ensure your applications specify a user agent if you still require use of the API after this date.

Please let me know if you have any questions.

Thanks,
James

1 Like

Hi @jamesevans I think you should allow more time particularly for devs using the unified api in their apps. Apps users are slow to update specially on Android and I can foresee bad ratings coming from users whose app suddenly stopped working. So please consider at least a couple of months or three for this kind of critical changes. Thank you.

@nakkore We would love to give a longer lead time but there is some urgency to get this change in place. I’ve contacted all the main transport apps that use our data and they are happy with the timeframe we’ve given them. Apologies to anyone else who is inconvenienced. We try our hardest to avoid changes that could break third party apps as we know our customers rely on them but in this instance we had no choice.

@jamesevans

For those of us who use the API as back-end processes, is there any particular thing you would like as a the user-interface parameter?

I think my personal default is to use User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 98)

Does using file_get_contents() and/or curl in PHP generates a “User-Agent” automatically?

Hi @briantist

As long as it’s populated, you should be OK. Sometimes we block particular user agents if there’s malicious or suspicious activity associated with it, but that’s quite rare.

In terms of PHP curls, I’m not sure what’s set at default, but you can specify the user agent in your curl command

Thanks
James

@jamesevans

Thanks. If there is no default User-Agent created by PHP, then I guess I’m going to have to contact MTR Crossrail and tell them their customer app is going to stop working…

Brian

@jamesevans

Sorry, I know someone is going to ask… is there a staging API server that we can use to test software against this change?

I’m not sure what will happen to existing code as I’m not expecting a 403 back, so didn’t code for it.

Agree, that ought to be done. I’m assuming firefox (desktop and android) will send a user agent unless I’ve told it not to, but I’m not sure how I would test it in both cases. A test server would confirm it properly.

@harry

Use Chrome’s developer tools. You can see the request headers by pressing F12.

Thanks Brian. I found almost exactly the same using F12 in firefox.

[And yes, it did show a user agent, so I suspect that will apply to anybody else using firefox].

@harry @briantist

I’m working on setting up a test domain for this. It’s proving tricky as it’s a global firewall setting for all our outward-facing web applications. I’ll let you know how I get on.

@briantist - curl on the command line does seem to send a user agent in the headers. You can see this by using curl -v

> GET /Line/85/Route HTTP/2

> Host: api.tfl.gov.uk

> User-Agent: curl/7.54.0

@jamesevans

It looks like PHP’s built-in CURL doesn’t add an it’s own user-agent. Here’s some code…

                $strURL = "https:/api.tfl.gov.uk/Line/85/Route";
            $strCacheFile = "tmp.saved";
            $ch = curl_init($strURL);
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
            $fh = fopen("$strCacheFile", "w");
            curl_setopt($ch, CURLOPT_FILE, $fh);
            curl_setopt($ch, CURLOPT_HEADER, 0);
            curl_setopt($ch, CURLOPT_TIMEOUT, 5);
            curl_setopt($ch, CURLOPT_VERBOSE, 5);
            curl_exec($ch);
            fclose($fh);
            curl_close($ch);

This is the output…

I note, however, that my standard CURL routine (in my “framework”) does include

 curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");

which shows up like this…

image

1 Like

You can specify a user agent using -A argument in curl too.

curl -A "Briantist" -v https://api.tfl.gov.uk/Line/150/Route

which produces the header:

> GET /Line/150/Route HTTP/2
> Host: api.tfl.gov.uk
> User-Agent: Briantist

@jamesevans

True, but I never use the command line in any of my code. PHP has always had a cURL system baked in https://www.php.net/manual/en/book.curl.php

Also, in PHP you can always use file_get_contents(“http://…”) to do any simple GET. https://www.php.net/manual/en/function.file-get-contents.php