DfT Bus Open Data

SJCooper · October 22, 2020, 9:15pm

Will TfL been providing data to the DfT’s Bus Open Data project?

TIA,

Simon

JakubGasiorowski · October 23, 2020, 4:21pm

Hi Simon,

Although TfL is exempt from DfT’s ‘The Public Service Vehicles (Open Data) (England) Regulations 2020’, that enforces bus operators to share bus real time location (and others), we are committed and are planning to release this data somewhere next year.

We will post on the Forum/Blog when we have more concrete information about releasing this dataset!

In the meantime, please have a look at our other Bus-related datasets available on our Open Data page:

Kind regards,
Jakub

briantist · October 24, 2020, 9:10am

Just for the info, if you want live bus information then the best place to get it from is the HTTP Digest interface at http://countdown.api.tfl.gov.uk/interfaces/ura/stream_V1

This is a bit odd in that it returns data in a CSV format, but it does push the data to the user.

The digest can be used with curl and so you get an event-based stream of data via “on_curl_write”.

I’ve used it for several projects now and it is very reliable.

briantist · October 24, 2020, 9:12am

Welcome @JakubGasiorowski

There is - for @SJCooper a downloadedable CSV file of all bus services here - https://data.tfl.gov.uk/tfl/syndication/feeds/bus-sequences.csv? app_id=eeeeeeee&app_key=111111111111111111111111

SJCooper · October 24, 2020, 12:36pm

Jakub / Brian,

Thanks for the replies. We’re looking forward to seeing what comes of TfL releasing data to the DfT feed. The use of block numbers in the DfT is data we’re often asked for which currently isn’t in the TfL feeds.

As background, I’ve been consuming the TfL Countdown feed for just over eight years now, originally translating the feed into enthusiast friendly Excel information and now as a minor part of the admin team behind lvf.io .

Have a good weekend,

Simon

mjcarchive · October 24, 2020, 1:05pm

If I could be allowed an idiot question, what are the “block numbers”?

SJCooper · October 24, 2020, 1:42pm

Running numbers…

SJCooper · August 2, 2021, 6:53pm

@JakubGasiorowski

Any update on this please?

Many thanks,

Simon

JakubGasiorowski · October 25, 2021, 10:46am

Hi Simon,

Apologies for no response earlier.

The recent update on this data is that it will be available via DfT’s BODS, rather than our Unified API. I don’t have any direct links yet, but DfT are working on getting this data live soon.

Kind regards,
Jakub

SJCooper · October 25, 2021, 5:29pm

Thanks for the update Jakub. I guess it’s in their beta system at the moment. Is it too early to say if the data via BODS will provide any other extra information such as bus loading info. / USB charging info that some other operators currently provide to BODS?

Simon

mjcarchive · October 28, 2021, 10:08am

Will it be mandatory for TfL operators to provide running/block number information to the DfT system? I ask because it is unlikely to be the chase for Fred’s Coaches of Upper Piddle as they operate two school journeys each day - and if it is not mandatory it quite likely won’t be provided. I tend to think of these numbers as a London thing but that is probably because they were only displayed on LT buses and their successors; they may be used internally. One large London operator actually claimed not to use running numbers for some years, arguing (I think) that it got in the way when managing the service and driver changeovers but that may have been spin!

Michael

SJCooper · October 28, 2021, 5:18pm

Hi Michael,

It’s now mandatory for all operators to provide running / block numbers to BODS, see How to use bus open data under “Bus location data” and ‘Block Ref’. For the timetables operators provide to BODS, BODS now audit the data and provide a “quality score”, one of the criteria here is the presence of block numbers, see Bus operator requirements . This link is to a Data Quality Report for a set of First Leeds timetables (which got 100% btw).
One of the things we’ve found for LVF is that actually having block refs in the location data match those given in the timetable TxCs is hit and miss though . I believe the TfL timetable TxC files do have the running numbers, so I’m really looking forward to seeing how these match up to the location data.

Regards,

Simon

mjcarchive · October 28, 2021, 5:24pm

Thanks, Simon. The TfL XML files on Datastore do not have block numbers AFAIK but of course its absence from one particular output says nothing about whether it is in the input.

Michael

JakubGasiorowski · October 29, 2021, 8:49am

Hi Simon,

We are not providing bus loading nor USB charging info in the feed, so it can’t be present in BODS. We don have a space in the feed for bus loading, but we don’t have the data coming in yet.
Hope this helps!

Kind regards,
Jakub

SJCooper · November 25, 2021, 8:54pm

Is this some TfL data in BODS we can see?

What’s with the mismatched LineRef and PublishedLineName?

briantist · November 26, 2021, 10:35am

@SJCooper I’m thinking you might find Technical guidance: SIRI-VM - GOV.UK the thing to answer these types of question?

john_wr · November 26, 2021, 12:37pm

Yes it is. We see >7000 new bus locations with OperatorRef=TFLO, they all refer to London agencies.
Btw do you know if the vehicle positions are available anywhere in TFL api now?

josh · November 27, 2021, 4:34am

(Not sure if this is the right place to report this) A few observations about the following bit of TfL Bus Open Data data:

<VehicleActivity>
    <RecordedAtTime>2021-11-27T04:16:04+00:00</RecordedAtTime>
    <ItemIdentifier>b27232b9-e7d8-47f6-a128-7676009847a3</ItemIdentifier>
    <ValidUntilTime>2021-11-27T04:21:58.915210</ValidUntilTime>
    <MonitoredVehicleJourney>
        <LineRef>443</LineRef>
        <DirectionRef>2</DirectionRef>
        <PublishedLineName>36</PublishedLineName>
        <OperatorRef>TFLO</OperatorRef>
        <OriginRef>490010204K</OriginRef>
        <OriginName>New Cross Bus Garage</OriginName>
        <DestinationRef>490012248W</DestinationRef>
        <DestinationName>Peckham Road / Southampton Way</DestinationName>
        <OriginAimedDepartureTime>2021-11-28T03:41:00+00:00</OriginAimedDepartureTime>
        <VehicleLocation>
            <Longitude>-0.165228</Longitude>
            <Latitude>51.516563</Latitude>
        </VehicleLocation>
        <Bearing>315.0</Bearing>
        <VehicleJourneyRef>81405</VehicleJourneyRef>
        <VehicleRef>9085</VehicleRef>
    </MonitoredVehicleJourney>
    <Extensions/>
</VehicleActivity>

The OriginAimedDepartureTime is one day in the future – seems to happen in the early hours of the morning
The VehicleRefs are different to the different to the vehicleIds used by the TfL API. It’s possible to convert between the two – 9085 above I think corresponds to YY67URO – but would be nicer if they were consistent
The DestinationRef and DestinationName aren’t quite right in the example above – the journey confinutes to Queen’s Park (490013353E)

mjcarchive · November 27, 2021, 9:37am

Might the behaviour of the destination data be sort of deliberate, with the next stop served being described (misleadingly) as such?

I am assuming that the final destination and other info (block/running number) are available but in a different chunk of data.

SJCooper · November 27, 2021, 12:17pm

Michael,

The BODS data isn’t meant to provide ‘Next Stop’ information which is what TfL seem to be doing here. This is hardly customer/passenger friendly “Is my bus going to the end of the route? Don’t know but it’s going to the next stop…”.
All the BODS data is meant to have is origin (and depending on which page you read for mandatory fields, maybe the destination - see below*). It’s one of the reasons why having the block ref (running number) is so key (and why BODS have made it a mandatory field (subtle hint TfL peeps ) - so that you can tie the trip up to the timetable (TxC) and work out from that what time the bus is due to serve the next stops. Unfortunately doing it this way, unlike the Countdown or Unified API, takes no account of known traffic delays and is a lot more CPU intensive on the server the developer hosts the system on.

Of note, unlike the TfL Unified or Countdown API, the data from BODS seems to include garage journeys… look for the all upper case origin or destination.

Josh,

Welcome to the TfL TechForum!
The VehicleRefs are the same as those provided in TfL’s Countdown API feed which also includes registration numbers. We’ve been using VehicleIDs over on LVF for nine years now…
I’m wondering if the date discrepancy is due to TfL’s services using a 28 hour clock for scheduling purposes?

Brian,

I understand the different between LineRef and PublishedLineName, what I don’t understand is why TfL have implemented it this way when all the other operators around the country have been sensible about it. If ‘regional op’ has two routes with the same public route number, let’s say “36” then in their data both will show PublishedLineName=“36” but the LineRefs may be “36” and “5036” so they can be differentiated - at least there’s some commonality as both include ‘36’. For TfL to use a LineRef which may match a completely different route is just odd in my opinion.
__

*AVL Data requirements from BODS for Bus Operators

from Bus operator requirements

BODS SIRI-VM data spec

(from How to use bus open data)

I’ve already complained to BODS about the mandatory data inconsistencies.

Simon