Processing data CSV files - couple of "gotchas" warnings

Unified API
1

Fellow Developers,

When processing the bus-stops.csv and bus-sequences.csv data files, ensure that you remove any extra “newline” character in the Naptan_Atco column. The Naptan_Atco field is very important when retrieving various data from the API, so it’s vital to store/use the correct value.

Here are a few examples:

101,1,42,BP6077,92152,"490018760E
",BECKTON DLR DEPOT,544383,181148,80,0

101,2,3,BP6078,92153,"490018760W
",BECKTON DLR DEPOT,544329,181128,260,0

BP5807,91387,"49009917N
",TOTTENHAM HALE STATION <> #,534465,189615,208,9J11,0

BP5809,91389,"1500420021Y0
",EPPING FOREST COLLEGE,543971,196285,253,UZ12,0

The data files are available here:
https://api-portal.tfl.gov.uk/docs

Happy and safe developing!!!

2

LondonTravel

Yeah, there in one of the live bus feeds too. Very annoying!

3

Another gotcha is a NAPTAN code of the form 490001089E4. If you are not careful some software will interpret this as a number in scientific notation, in this case 4900010890000, or as it may appear, 4.90E+12. There arenpt many of these - mainly in the Croydon and Sutton areas - but they are a pain.

I have a vague memory of a similar interpretation issue with codes ending in (say) F4 but cannot remember the details. There would be far less of these because E is used much more often as a qualifier than F.

Michael

4

Hi Brian,

Thanks for the heads up!

Regarding the live bus feeds - as I am a newbie - where can I get documentation on plugging into the live feed using the websocket?

Thanks,
Winston.

1 Like
5

Hi Michael,

My advice is to read and store it in a string format. I know some applications will “cleverly” (automatically) interpret it as a “number” format because there are no quotes - at least you can override this default behaviour.

Anyway…thanks for sharing your pain and…others…beware!

Regards,
Winston.

1 Like
6

I think the iBus stuff (not the unified API, which gives snapshots) is doced here

The main iBus “live” feed is here

http://countdown.api.tfl.gov.uk/interfaces/ura/stream_V1

You’ll need some special credentials . See https://api-portal.tfl.gov.uk/docs - scroll down to Bus, Coach and River and then see the second (“stream”) link. It is all documented, but it’s much more fun to guess!

7

Here’s how I did it PHP (for the digest mode)

class digestCommon
{
const DEBUGMODE = false;
const LINEID = "UL16,UL51,90";
public $isAlreadyRunning = null;


public function setAlreadyRunning($isAlreadyRunning)
{
    $this->isAlreadyRunning = $isAlreadyRunning;
}

public function startDigest($strURL = "http://countdown.api.tfl.gov.uk/interfaces/ura/stream_V1",
                            $strUsername = "***********",
                            $strPassword = "***********",
                            $arrPostParameters = ["LineID" => self::LINEID,
                                "ReturnList" => "StopCode2,StopPointIndicator,DestinationName,EstimatedTime,ExpireTime,VehicleID,StopID"], $intAuth = CURLAUTH_DIGEST, $strFunctionName = "on_curl_write")
{
    if (self::DEBUGMODE) {
        error_reporting(E_ALL);
        ini_set('display_errors', '1');
    }
    if (is_array($arrPostParameters)) {
        $arrPostParameters = http_build_query($arrPostParameters);
    }
    echo PHP_EOL . __FUNCTION__ . " is sending " . strlen(var_export($arrPostParameters, true)) . " bytes to {$strURL}";
    echo PHP_EOL . PHP_EOL . PHP_EOL;
    $arrOptions = [CURLOPT_URL => $strURL,
        CURLOPT_HEADER => false,
        CURLOPT_VERBOSE => self::DEBUGMODE,
        CURLOPT_RETURNTRANSFER => $strFunctionName !== "",
        CURLOPT_FOLLOWLOCATION => true,
        CURLOPT_SSL_VERIFYPEER => substr($strURL, 0, 5) === "https",    // for https
        CURLOPT_USERPWD => $strUsername . ":" . $strPassword,
        CURLOPT_HTTPAUTH => $intAuth,
        CURLOPT_POST => true,
        CURLOPT_POSTFIELDS => $arrPostParameters];
    if ($strFunctionName != "") {
        $arrOptions = $arrOptions + [CURLOPT_WRITEFUNCTION => array($this, $strFunctionName)];
    }
    $resCurlHandle = curl_init();
    curl_setopt_array($resCurlHandle, $arrOptions);
    try {
        $RawResponce = curl_exec($resCurlHandle);
        // validate CURL status
        if (curl_errno($resCurlHandle)) {
            throw new \Exception(curl_error($resCurlHandle), 500);
        }
        // validate HTTP status code (user/strPassword credential issues)
        $intStatusCode = curl_getinfo($resCurlHandle, CURLINFO_HTTP_CODE);
        if ($intStatusCode != 200) {
            throw new \Exception("Response with Status Code [" . $intStatusCode . "].", 500);
        }
    } catch (\Exception $ex) {
        if ($resCurlHandle != null) {
            curl_close($resCurlHandle);
        }
        throw new \Exception($ex);
    }
    if ($resCurlHandle !== null) curl_close($resCurlHandle);
    return $RawResponce;
}
}
8

Hi Brian,

This is mighty of you!

Let me get cracking…will share with you when I have something presentable.

Respect,
Winston.