NeTEx & SIRI

Show me the code

Alban Peignier

February 2, 2025

Introduction

NeTEx and SIRI are described as complex and painful.

But what the matter if they only require a few lines of code ?

Who I am

Alban Peignier - @apeignier@mamot.fr

CPO & Engineer at enRoute

Providing Chouette SaaS & Ara SaaS

Free software for mobility data

Agenda

  • Read NeTEx file
  • Invoke a SIRI API
  • Write NeTEx file
  • Create a SIRI Server
  • What’s missing ?

Who are you ?

  • Who uses GTFS ?
  • Who has ever seen a NeTEx file ?
  • Made code to read a NeTEx file ?
  • Made code to create a NeTEx file ?
  • Invoked a SIRI API ?

Read a NeTEx file

source = Netex::Source.read('sample.zip')

source.stop_places.each do |stop_place|
  puts stop_place.name
end

Read a NeTEx file

Works on any file

XML, ZIP, any “profile”, any producer

A small XML sample or an archive of 10 Giga bytes

Even with an invalid file

Read a NeTEx file

As easy as in GTFS

Richer, more complete “models”

<StopPlace id="42">
  <Name>Gare Routière</Name>
  <Centroid>
    <Location>
     <Longitude>-4.618419</Longitude>
     <Latitude>48.4323529</Latitude>
    </Location>
  </Centroid>
  <PostalAddress>
    <AddressLine1>8 Rue du Pont de Bois</AddressLine1>
    <Town>Saint-Renan</Town>
    <PostCode>29290</PostCode>
  </PostalAddress>
  <placeTypes>
    <TypeOfPlaceRef ref='multimodalStopPlace'/>
  </placeTypes>
  <TransportMode>bus</TransportMode>
  <StopPlaceType>onstreetBus</StopPlaceType>
  <AccessibilityAssessment version="any" id="test">
    <MobilityImpairedAccess>true</MobilityImpairedAccess>
    <limitations>
      <AccessibilityLimitation>
        <WheelchairAccess>true</WheelchairAccess>
        <StepFreeAccess>true</StepFreeAccess>
        <EscalatorFreeAccess>true</EscalatorFreeAccess>
        <LiftFreeAccess>true</LiftFreeAccess>
        <AudibleSignalsAvailable>true</AudibleSignalsAvailable>
        <VisualSignsAvailable>false</VisualSignsAvailable>
      </AccessibilityLimitation>
    </limitations>
  </AccessibilityAssessment>
</StopPlace>

Read a NeTEx file

stop_place.name
stop_place.latitude
stop_place.longitude
stop_place.postal_address
stop_place.accessibility_assessment
# ...
stop_place.quays
stop_place.key_lists

Seems to easy ?

NeTEx allows variety …

A lot of variety

And Code doesn’t like variety

(NeTEx + xpath = epic fail)

NeTEx Variety

in Archive file

Large ZIP file

Archive:  publications-netex-du-24-10-2024.zip
Name
----
OFFRE_7bb20cb9-bb2b-4bf8-beb1-06b68a5615f9_Poissy_-_Les_Mureaux/offre_6fc38810-b245-438d-98a7-dd1e0be38124_91.xml
OFFRE_7bb20cb9-bb2b-4bf8-beb1-06b68a5615f9_Poissy_-_Les_Mureaux/offre_2222cffa-9430-499a-9a82-bc1df79ffd9d_41.xml
...
OFFRE_dd8ea746-43fd-4566-ac20-98eb8f2ad1dd_Keolis_Roissy_Pays_de_France_Ouest/calendriers.xml
lignes.xml
arrets.xml
-------
2052 files

NeTEx Variety

in Archive file

… Or a large unique XML

Each “Profile” creates its own structure

NeTEx Resources are never “at the same place”

😥

NeTEx Variety

into the XML Files

Verbose XML files

NeTEx Profiles love “Frames”

<PublicationDelivery xmlns="http://www.netex.org.uk/netex">
  <PublicationTimestamp>2024-11-06T07:08:59Z</PublicationTimestamp>
  <ParticipantRef>enRoute</ParticipantRef>
  <dataObjects>
    <CompositeFrame id="FR:CompositeFrame:NETEX_LIGNE:LOC">
      <Name>Airport - Bullfrog</Name>
      <TypeOfFrameRef ref="FR:TypeOfFrame:NETEX_LIGNE:"/>
      <frames>
        <GeneralFrame id="FR:GeneralFrame:NETEX_LIGNE:LOC">
          <TypeOfFrameRef ref="FR:TypeOfFrame:NETEX_LIGNE:"/>
          <members>

NeTEx Variety

Code is solution

What’s matter in a NeTEx file ? NeTEx ressources ?

source.stop_places
source.lines
source.service_journeys
source.point_of_interests

✅ Direct access to NeTEx resources

NeTEx Variety

Code is solution

Forget file structure

Frame, file, line, column via tags

source.lines.first.tags
=> {>"lignes.xml", >13, >13, 
    >"FR:GeneralFrame:NETEX_LIGNE:LOC", ...

NeTEx Variety

Code is solution

File structure is optional

… at least for code

xml = <<XML
<StopPlace>
  <Name>Simple Sample</Name>
</StopPlace>
XML

source.parse(StringIO.new(xml))
puts source.stop_places.first.name
=> "Simple Sample"

NeTEx Variety

Into NeTEx Resources

NeTEx Resources / “Models” are defined by the NeTEx specs

Very few varieties at this level

But much more complex to manage by code

NeTEx Variety

Into NeTEx Resources

Small example

<Centroid>
  <Location>
    <Longitude>-4.618419</Longitude>
    <Latitude>48.4323529</Latitude>
  </Location>
</Centroid>
<Centroid>
  <Location>
    <gml:pos srsName="EPSG:27572">613081 2410342</gml:pos>
  </Location>
</Centroid>

NeTEx Variety

Code is solution

Transformers

Process resources to return the expected form

source.transformers <<
  Netex::Transformer::LocationFromCoordinates.new

Invoke a SIRI API

response = client.send SIRI::LinesDiscovery::Request.new
line_id = response.annotated_line_refs.sample.line_ref

request = SIRI::EstimatedTimetable::Request.new.tap do |request|
  request.lines << SIRI::LineDirection.new(line_id)
end

puts client.send request

A real API, not just a binary file

Invoke a SIRI API

response = client.send SIRI::StopPointsDiscovery::Request.new
stop_area_id =
  response.annotated_stop_point_refs.sample.stop_point_ref

request = SIRI::StopMonitoring::Request.new(
  stop_area_id
)

response = client.send request
puts response.monitored_stop_visits.first&.expected_departure_time

SIRI Variety

The protocol “setup”

SIRI uses HTTP request and:

  • XML “raw” with POST
  • XML SOAP
  • JSON with Restful API

But Protocol definition and structures are the same

SIRI Variety

Code is solution

response = client.send SIRI::CheckStatus::Request.new

Code is generic, whatever the protocol setup

request = SIRI::CheckStatus::Request.new

SIRI::Raw::Builder.for(request).to_payload
SIRI::SOAP::Builder.for(request).to_payload
SIRI::JSON::Builder.for(request).to_payload

Receive SIRI live notifications

Subscribe

request = SIRI::Subscribe::Request.new
subscription = 
  SIRI::Subscribe::Request::Subscription::VehicleMonitoring.new(
    credential,
    SecureRandom.uuid,
    Time.now + 3600,
  )
request.subscription_requests << subscription

response = client.send request

Receive SIRI live notifications

Receive Notifications

server.receive(SIRI::VehicleMonitoring::Notification) do |n|
  n.vehicle_activities.each do |vehicle_activity|
    description = attributes.map do |attribute|
      vehicle_activity.send(attribute) || '-'
    end.join(' ')

    puts "🚌 #{description}"
  end

  SIRI::DataReceivedAcknowledgement.new true
end

Receive SIRI live notifications

🥳 Enjoy

🚌 51 47.85724615430222 1.9050266812693997 174.0 - A 11_A06_6_18:50:00 2025-02-01 18:25:57 UTC
🚌 52 47.89505287724359 1.904487986758294 358.0 - A 11_A14_6_18:53:30 2025-02-01 18:25:57 UTC
🚌 45 47.92644341158054 1.9070538704177726 343.0 - A 11_A02_6_18:37:30 2025-02-01 18:25:57 UTC
🚌 60 47.839699622076736 1.9348941865508855 24.0 - A 11_A11_6_19:18:00 2025-02-01 18:25:57 UTC
🚌 48 47.87681771498911 1.9122357153186686 151.0 - A 11_A07_6_18:58:00 2025-02-01 18:25:57 UTC
🚌 78 47.846057734606006 1.9175851740217948 304.0 - A 11_A04_6_19:09:30 2025-02-01 18:25:57 UTC
🚌 42 47.854589704018096 1.9087590153289151 129.0 - A 11_A12_6_18:42:00 2025-02-01 18:25:57 UTC
🚌 40 47.9141923657212 1.9011074285183922 150.0 - A 11_A08_6_19:14:00 2025-02-01 18:25:57 UTC
🚌 50 47.874828005835255 1.9131460393913398 355.0 - A 11_A03_6_19:01:30 2025-02-01 18:25:57 UTC
🚌 43 47.928184457723255 1.9250431206818681 270.0 - A 11_A09_6_19:30:00 2025-02-01 18:23:57 UTC
🚌 39 47.83998941602201 1.9350889202646175 207.0 - A 11_A15_6_18:34:00 2025-02-01 18:25:57 UTC
🚌 73 47.90102296279369 1.9028523233359471 165.0 - A 11_A01_6_19:06:00 2025-02-01 18:25:57 UTC
🚌 58 47.928146431321046 1.9189373270217 271.0 - A 11_A13_6_19:22:00 2025-02-01 18:25:57 UTC
🚌 57 47.83736531991079 1.9188182936256228 250.0 - A 11_A05_6_19:26:00 2025-02-01 18:25:57 UTC
🚌 41 47.908108492305004 1.904044545180632 5.0 - A 11_A10_6_18:45:30 2025-02-01 18:25:57 UTC
🚌 68 47.90188886014469 1.8982433698255263 272.0 - B 11_B02_6_18:59:15 2025-02-01 18:25:57 UTC
🚌 65 47.90027188750417 1.8842766003856504 79.0 - B 11_B11_6_19:15:00 2025-02-01 18:25:57 UTC
🚌 103 47.90945360504696 1.94377577360228 39.0 - O 11_O01_6_19:03:00 2025-02-01 18:18:57 UTC
🚌 106 47.90697549483362 1.9060892641365008 286.0 - O 11_O02_6_19:13:00 2025-02-01 18:25:57 UTC
🚌 104 47.899214821788455 1.9048839512844586 178.0 - O 11_O03_6_19:23:00 2025-02-01 18:25:57 UTC

Seems to easy ?

Indeed …

Inaccuracies create local interpretations
mostly on advanced points

Works in progress to identify and resolve them

How to find NeTEx datasets ?

How to find SIRI APIs ?

Free and open SIRI servers are too rare :(

An urban legend says that it is complicated

  • Norway enTur SIRI APIs
  • French NAP recommends public server
    • with “open-data” RequestorRef
  • Our Ara SaaS users can setup public APIs in two clicks. Ask them !

Create a NeTEx file

One is missing ?

target = Netex::Target.build("sample.zip")

target << Netex::StopPlace.new(..)
target << Netex::Line.new(...)
target << Netex::Route.new(...)
target << Netex::ServiceJourneyPattern.new(...)
target << Netex::ServiceJourney.new(...)

target.close

Create a NeTEx file

How to manage profile ?

profile = Netex::Profile::French.new
target = Netex::Target.build("sample.zip", profile)

target << Netex::StopPlace.new(..)
# ... 
# Same code

Create a file structured as expected by the French NeTEx Profile

Create a NeTEx file

How to create a profile ?

class MyProfile < Netex::Profile::Base
  # Route NeTEx resources to the expected document
  def document_for(resource)
    case resource
      when Netex::StopPlace, Netex::Quay, Netex::StopPlaceEntrance
        document(Document::Stops)
      when Netex::DayType, ...
        document(Document::Calendars)
      when Netex::Line, Netex::Route, ...
        document(Document::Line, resource.tag[])
    # ...
  end
end

Create a NeTEx file

How to create a profile ?

  # Define specific XML file
class Document::Stops < Netex::Target::Document
  def initialize
    super "stops.xml"

    frames << general_frame('STOPS') do |frame|
      frame.sections.create Netex::StopPlace
      frame.sections.create Netex::Quay
      frame.sections.create Netex::StopPlaceEntrance
    end
  end
end

Create a SIRI server

I’m alive:

server = SIRI::Server.new do |s|
  s.use SIRI::Server::Marshall, SIRI::Raw
  s.use SIRI::Server::Timestamp
  s.use SIRI::Server::MessageIdentifier
end

server.receive(SIRI::CheckStatus::Request) do
  SIRI::CheckStatus::Response.new(
    service_started_time, 
    true
  )
end

Create a SIRI server

Respond to your first EstimatedTimetable request:

server.receive(SIRI::EstimatedTimetable::Request) do |request|
  SIRI::EstimatedTimetable::Response.new.tap do |response|
    journey = SIRI::EstimatedVehicleJourney.new

    # For the (near) past
    journey.recorded_calls << SIRI::Call.new(...)

     # For the future
    journey.estimated_calls << SIRI::Call.new(...)
    response.estimated_vehicle_journeys << journey
  end
end

Our open source libraries

Mostly in Ruby, use in production

🚧 Under construction:

Our NeTEx SaaS solutions

Don’t reinvent the wheel

Validate a NeTEx file … with your own rules
with Chouette Valid

Convert NeTEx to NeTEx / to GTFS
with Chouette Convert

What’s missing ?

SIRI & NeTEx Developer resources

Open source generic libraries in other languages❔
(🙏stop generating code from the XSD)

Tools and UX for NeTEx and SIRI documentation
and National Profiles documentations

More Developers in European and National Workgroups

Questions