TfL rail "vehicles"

Using the new unified api, I had a look at extending my bus arrivals software for rail traffic as well. I came across two strange things using TfL-rail as the Line-ID that I wondered if anyone had an answer to?

  1. The Vehicle-ID field has what seems to be a random string of characters, always ending in ==, for example “3MdUUvZiXNeuLNbJa31GEw==”. These can be seen by using the api with something like /Line/TfL-rail/Arrivals/910GLIVST Does anyone know if this is interpretable (for buses it is more often or not the vehicle registration, and for trams it is the 4-digit fleet number)?

  2. When using the vehicle-ID in the predictions reference (I.e. /vehicle/3MdUUvZiXNeuLNbJa31GEw==/arrivals ) it only gives one station and not more down the line. The example I gave was giving Forest Gate on its way to Liverpool Street, but did not give Maryland, Stratford or Liverpool Street, all of which would be well within the 30 minutes supplied by the api. Does anyone know why no more stops are shown for this “vehicle”?

I understand these are quite specific questions, and may need someone from TfL to answer them.

I’m not a TfL person, but we can do some thinking amongst ourselves while we wait for someone to reply.

So that random string of characters is probably base64 encoded - I say base64 rather than base62 because there is currently a / in one of them: HFBB4vec1exFgIE/onkaKw==, and base64 uses + and / as additional characters. Unfortunately decoding this as UTF-8 gives us PAE?y+. and none of the other common encodings give anything clearer.

Maybe it means something more sensible if you look at in binary?

Hi Busspotter,

The VehicleID result we provide passes the serviceID result from Darwin (National Rail API.

Their API documentation gives the following:

serviceID The unique service identifier of this service relative to the station board on which it is displayed. This value can be passed to GetServiceDetails to obtain the full details of the individual service.

When calling their API with a serviceID, you do get both the previous calling points of the service (and their timings) as well as the subsequent calling points and timings.

I’ll do some more digging on our side to see why you’re seeing only certain stations on the vehicle endpoint.

<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
   <soap:Body>
      <GetServiceDetailsResponse xmlns="http://thalesgroup.com/RTTI/2014-02-20/ldb/">
         <GetServiceDetailsResult xmlns:lt3="http://thalesgroup.com/RTTI/2015-05-14/ldb/types" xmlns:lt5="http://thalesgroup.com/RTTI/2016-02-16/ldb/types" xmlns:lt4="http://thalesgroup.com/RTTI/2015-11-27/ldb/types" xmlns:lt="http://thalesgroup.com/RTTI/2012-01-13/ldb/types" xmlns:lt2="http://thalesgroup.com/RTTI/2014-02-20/ldb/types">
            <lt2:generatedAt>2017-06-27T10:38:39.6840557+01:00</lt2:generatedAt>
            <lt2:serviceType>train</lt2:serviceType>
            <lt2:locationName>Maryland</lt2:locationName>
            <lt2:crs>MYL</lt2:crs>
            <lt2:operator>TFL Rail</lt2:operator>
            <lt2:operatorCode>XR</lt2:operatorCode>
            <lt2:platform>1</lt2:platform>
            <lt2:sta>10:36</lt2:sta>
            <lt2:ata>On time</lt2:ata>
            <lt2:std>10:36</lt2:std>
            <lt2:atd>10:38</lt2:atd>
            <lt2:previousCallingPoints>
               <lt2:callingPointList serviceType="train" serviceChangeRequired="false" assocIsCancelled="false">
                  <lt2:callingPoint>
                     <lt:locationName>Shenfield</lt:locationName>
                     <lt:crs>SNF</lt:crs>
                     <lt:st>10:04</lt:st>
                     <lt:at>On time</lt:at>
                  </lt2:callingPoint>
                  <lt2:callingPoint>
                     <lt:locationName>Brentwood</lt:locationName>
                     <lt:crs>BRE</lt:crs>
                     <lt:st>10:07</lt:st>
                     <lt:at>On time</lt:at>
                  </lt2:callingPoint>
                  <lt2:callingPoint>
                     <lt:locationName>Harold Wood</lt:locationName>
                     <lt:crs>HRO</lt:crs>
                     <lt:st>10:12</lt:st>
                     <lt:at>On time</lt:at>
                  </lt2:callingPoint>
                  <lt2:callingPoint>
                     <lt:locationName>Gidea Park</lt:locationName>
                     <lt:crs>GDP</lt:crs>
                     <lt:st>10:16</lt:st>
                     <lt:at>On time</lt:at>
                  </lt2:callingPoint>
                  <lt2:callingPoint>
                     <lt:locationName>Romford</lt:locationName>
                     <lt:crs>RMF</lt:crs>
                     <lt:st>10:18</lt:st>
                     <lt:at>On time</lt:at>
                  </lt2:callingPoint>
                  <lt2:callingPoint>
                     <lt:locationName>Chadwell Heath</lt:locationName>
                     <lt:crs>CTH</lt:crs>
                     <lt:st>10:22</lt:st>
                     <lt:at>On time</lt:at>
                  </lt2:callingPoint>
                  <lt2:callingPoint>
                     <lt:locationName>Goodmayes</lt:locationName>
                     <lt:crs>GMY</lt:crs>
                     <lt:st>10:24</lt:st>
                     <lt:at>On time</lt:at>
                  </lt2:callingPoint>
                  <lt2:callingPoint>
                     <lt:locationName>Seven Kings</lt:locationName>
                     <lt:crs>SVK</lt:crs>
                     <lt:st>10:26</lt:st>
                     <lt:at>On time</lt:at>
                  </lt2:callingPoint>
                  <lt2:callingPoint>
                     <lt:locationName>Ilford</lt:locationName>
                     <lt:crs>IFD</lt:crs>
                     <lt:st>10:29</lt:st>
                     <lt:at>On time</lt:at>
                  </lt2:callingPoint>
                  <lt2:callingPoint>
                     <lt:locationName>Manor Park</lt:locationName>
                     <lt:crs>MNP</lt:crs>
                     <lt:st>10:32</lt:st>
                     <lt:at>On time</lt:at>
                  </lt2:callingPoint>
                  <lt2:callingPoint>
                     <lt:locationName>Forest Gate</lt:locationName>
                     <lt:crs>FOG</lt:crs>
                     <lt:st>10:34</lt:st>
                     <lt:at>On time</lt:at>
                  </lt2:callingPoint>
               </lt2:callingPointList>
            </lt2:previousCallingPoints>
            <lt2:subsequentCallingPoints>
               <lt2:callingPointList serviceType="train" serviceChangeRequired="false" assocIsCancelled="false">
                  <lt2:callingPoint>
                     <lt:locationName>Stratford (London)</lt:locationName>
                     <lt:crs>SRA</lt:crs>
                     <lt:st>10:38</lt:st>
                     <lt:et>On time</lt:et>
                  </lt2:callingPoint>
                  <lt2:callingPoint>
                     <lt:locationName>London Liverpool Street</lt:locationName>
                     <lt:crs>LST</lt:crs>
                     <lt:st>10:47</lt:st>
                     <lt:et>On time</lt:et>
                  </lt2:callingPoint>
               </lt2:callingPointList>
            </lt2:subsequentCallingPoints>
         </GetServiceDetailsResult>
      </GetServiceDetailsResponse>
   </soap:Body>
</soap:Envelope>

@jamesevans is correct - for this feed we populate the vehicleId with the serviceId from Darwin. However, the way our /Vehicle endpoint works is that we simply return all arrivals predictions from our database tagged with that vehicleId - we don’t call back to Darwin’s GetServiceDetails.

The assumption is that we would already have all of the predictions for that vehicle in our database.

I can see straight away that by using the serviceId as VehicleId without cleansing it, its makes it tricky to use in /Vehicle/{vehicleId}/Arrivals - I’ll raise a ticket to resolve that. SVC-3940

We’ll also investigate why we’re not seeing predictions for the rest of the route… I suspect it’s either a retention period or because we don’t have the full national rail dataset loaded as reference data. SVC-3491.

I have a further comment about this, which confuses me no end - perhaps someone has an idea on it.

The vehicle ID returned by the TfLRail Line arrivals data (as shown in point 1 in my first message), often returns a “/” character in the vehicle ID - e.g. just now the following was returned:

{“$type”:“Tfl.Api.Presentation.Entities.Prediction,Tfl.Api.Presentation.Entities”,
“id”:“1666194400”,
“operationType”:1,
“vehicleId”:“nRVquwihWrqj/mjSj0Ahmg==”,
“naptanId”:“910GLIVST”,
“stationName”:“London Liverpool Street Rail Station”,
“lineId”:“tfl-rail”
etc…

I have tried several options including the following,
https://api.tfl.gov.uk/Vehicle/nRVquwihWrqj/mjSj0Ahmg==/Arrivals
https://api.tfl.gov.uk/Vehicle/nRVquwihWrqj//mjSj0Ahmg==/Arrivals
https://api.tfl.gov.uk/Vehicle/nRVquwihWrqj\/mjSj0Ahmg==/Arrivals
https://api.tfl.gov.uk/Vehicle/nRVquwihWrqj%2FmjSj0Ahmg==/Arrivals
https://api.tfl.gov.uk/Vehicle/nRVquwihWrqj%2F%2FmjSj0Ahmg==/Arrivals
https://api.tfl.gov.uk/Vehicle/“nRVquwihWrqj/mjSj0Ahmg==”/Arrivals
https://api.tfl.gov.uk/Vehicle/“nRVquwihWrqj%2FmjSj0Ahmg==”/Arrivals

but they all return the following error (with various degrees of the interpretation of the “/” character

{“$type”:“Tfl.Api.Presentation.Entities.ApiError, Tfl.Api.Presentation.Entities”,
“timestampUtc”:“2017-09-11T09:49:41.2242581Z”,
“exceptionType”:“EntityNotFoundException”,
“httpStatusCode”:404,
“httpStatus”:“NotFound”,
“relativeUri”:“/Vehicle/nRVquwihWrqj/mjSj0Ahmg==/Arrivals”,
“message”:“Resource not found: http://api:8001/Vehicle/nRVquwihWrqj/mjSj0Ahmg==/Arrivals
}

Any ideas how I can therefore use the vehicle ID as quoted?

This is getting daft now - the TfLRail API data is totally screwed. I just did the following enquiry, which should return every arrival on the TfL Rail line between Shenfield and Liverpool Street, and as far as I know, no other lines:

https://api.tfl.gov.uk/mode/tflrail/arrivals

However, the first few arrivals are as follows:

[
{
“$type”:“Tfl.Api.Presentation.Entities.Prediction, Tfl.Api.Presentation.Entities”,
“id”:”-1551819708”,
“operationType”:1,“vehicleId”:"pq2tWeKchumJSGhDC7Ep5A==“,
“naptanId”:"910GGOSPLOK”,
“stationName”:"Gospel Oak Rail Station”,
“lineId”:"tfl-rail”,
“lineName”:“TfL Rail”,
“platformName”:“Unknown”,
“direction”:”“,
“bearing”:”“,
“destinationNaptanId”:"910GSHENFLD”,
“destinationName”:“Shenfield Rail Station”,
“timestamp”:“2017-09-11T10:18:54Z”,
“timeToStation”:21,“currentLocation”:”“,
“towards”:”“,
“expectedArrival”:"2017-09-11T10:19:15Z”,
“timeToLive”:"2017-09-11T10:19:15Z”,
“modeName”:"tflrail”,
“timing”:{
“$type”:"Tfl.Api.Presentation.Entities.PredictionTiming, Tfl.Api.Presentation.Entities”,
“countdownServerAdjustment”:"00:00:00”,
“source”:"0001-01-01T00:00:00”,
“insert”:"0001-01-01T00:00:00”,
“read”:"0001-01-01T00:00:00Z”,
“sent”:"2017-09-11T10:18:54Z”,
“received”:“0001-01-01T00:00:00”
}
},
{
“$type”:“Tfl.Api.Presentation.Entities.Prediction, Tfl.Api.Presentation.Entities”,
“id”:”-1423392052”,
“operationType”:1,“vehicleId”:"pq2tWeKchumJSGhDC7Ep5A==“,
“naptanId”:"910GEMRSPKH”,
“stationName”:"Emerson Park Rail Station”,
“lineId”:"tfl-rail”,
“lineName”:“TfL Rail”,
“platformName”:“4”,
“direction”:”“,
“bearing”:”“,
“destinationNaptanId”:"910GSHENFLD”,
“destinationName”:“Shenfield Rail Station”,
“timestamp”:“2017-09-11T10:18:54Z”,
“timeToStation”:24,“currentLocation”:”“,
“towards”:”“,
“expectedArrival”:"2017-09-11T10:19:18Z”,
“timeToLive”:"2017-09-11T10:19:18Z”,
“modeName”:"tflrail”,
“timing”:{
“$type”:"Tfl.Api.Presentation.Entities.PredictionTiming, Tfl.Api.Presentation.Entities”,
“countdownServerAdjustment”:"00:00:00”,
“source”:"0001-01-01T00:00:00”,
“insert”:"0001-01-01T00:00:00”,
“read”:"0001-01-01T00:00:00Z”,
“sent”:"2017-09-11T10:18:54Z”,
“received”:“0001-01-01T00:00:00”
}
},
{
“$type”:"Tfl.Api.Presentation.Entities.Prediction, Tfl.Api.Presentation.Entities”,
“id”:"868586617”,
“operationType”:1,“vehicleId”:"pq2tWeKchumJSGhDC7Ep5A==“,
“naptanId”:"910GHGHMSPK”,
“stationName”:"Highams Park Rail Station”,
“lineId”:"tfl-rail”,
“lineName”:“TfL Rail”,
“platformName”:“4”,
“direction”:”“,
“bearing”:”“,
“destinationNaptanId”:"910GSHENFLD”,
“destinationName”:“Shenfield Rail Station”,
“timestamp”:“2017-09-11T10:18:54Z”,
“timeToStation”:24,“currentLocation”:”“,
“towards”:”“,
“expectedArrival”:"2017-09-11T10:19:18Z”,
“timeToLive”:"2017-09-11T10:19:18Z”,
“modeName”:"tflrail”,
“timing”:{
“$type”:"Tfl.Api.Presentation.Entities.PredictionTiming, Tfl.Api.Presentation.Entities”,
“countdownServerAdjustment”:"00:00:00”,
“source”:"0001-01-01T00:00:00”,
“insert”:"0001-01-01T00:00:00”,
“read”:"0001-01-01T00:00:00Z”,
“sent”:"2017-09-11T10:18:54Z”,
“received”:“0001-01-01T00:00:00”
}
}

You can see from this that the vehicle ID is the same, however, this train seems to be going from Gospel Oak at 10:19:15Z, then to Emerson Park at 10:19:18Z, then to Highams Park at 10:19:18Z. It has an eventual destination of Shenfield. Normally, Gospel Oak to Emerson Park would take over 1 hour and three trains - it would be lovely to be able to get on a train at Gospel Oak and three seconds later get off at Emerson Park. It would also be great to have a direct service from both places to Shenfield! As for stepping on the train at Emerson Park and then stepping straight back off again and finding yourself at Highams Park, even the likes of Doctor Who couldn’t manage that one!!!

The rest of the data, without all the “noise” shows trains that have a final destination of either Shenfield or Liverpool Street from Denmark Hill, Dalston Kingsland, Camden Road, Hamstead Heath, Goodmayes, Finchley Road and Frognal, Forest Gate, Richmond, Crouch Hill, Honor Oak Park, Clapton, Kensington Olympia, Kilburn High Road, Chadwell Heath, Kensal Rise, Kensal Green, Harold Wood, Manor Park, Peckham Rye, Norwood Junction, Hackney Wick, Cheshunt, Acton Central, Shenfield, Leyton Midland Road, Caledonian Road and Barnsbury, Harrow & Wealdstone, Brondesbury, Shadwell, Carpenters Park, South Tottenham, Stonebridge Park, and Hoxton.

There is something clearly wrong here, and this may explain why my original query of the vehicle ID is happening - because the data is corrupt and meaningless, the data is unable to be used in its current form.

Hi @Busspotter thanks for your feedback. We are still investigating these known issues with the format of the ids for TfL Rail vehicles.

Tim - it’s now been 7 months since I raised the original query, and 4 months since your last update. Did you manage to get anywhere on any of this TfL Rail stuff, or are you waiting until the Elizabeth Line is up and running, when hopefully the problem will go away (or will it just get worse with a mixture of Network Rail at either end and TfL in the tunnel section in the middle?!!)

Hi Busspotter,

From Tim’s comment above (Jun '17) ‘I can see straight away that by using the serviceId as VehicleId without cleansing it, its makes it tricky to use in /Vehicle/{vehicleId}/Arrivals - I’ll raise a ticket to resolve that. SVC-3940’ has been resolved.

Regarding LO train predictions appearing in TfL Rail StopPoints (SVC-4266) and TfL Rail train predictions appearing in LO StopPoints, (SVC-4957) we hope to have this delivered in the next upcoming sprint.

Dave