Execution model¶

In Sen your code has an entry point. This means that the rate at which it runs can be controlled, and therefore you can switch between real-time and stepped execution (which can be slower or faster than real-time).

Determinism¹ is the idea that what happens next follows entirely from what came before. In this context that becomes:

The next state is determined completely by the computations based on the current state.

This idea of "current" and "next" state is built into the Sen architecture and great care has been put into enabling it. In fact, all objects are double buffered and proxy objects (local and remote) are created to allow it.

Note: Sen will enable deterministic behavior, but you need to do your part. For example, if you use random numbers, do not seed your generator with the current time. Instead, make the seed part of the input parameters used during initialization.

The same care applies to anything else that varies between runs. A subscription hands you its objects in the order they arrived, which is stable within a process and not guaranteed across them, so a calculation that sums over a list is sensitive to that order in exactly the way floating-point addition is. Sen's own threading does not add to this: each component runs on its own thread, but every one of them reads the same frozen snapshot, so the risk comes from threads you start yourself and from arithmetic that depends on order.

The drain-update-commit cycle¶

To achieve determinism, Sen components execute in iterations. In each iteration they:

1. Drain their inputs to get a snapshot of the outside world. This "picture" does not change during the execution cycle. Not even time changes. The world is effectively frozen. You don't have to worry about this step.
2. Perform the needed computations and produce outputs. Outputs are write-only during this stage. This means that the values that you set and the events that you produce will not be visible (not even by you) until you commit your outputs. This is the step that you worry about.
3. Commit the outputs, so that they become visible to other components in a self-consistent manner. You also don't have to worry about this step.

This is the drain-update-commit approach, and you can think of it as frames in a movie:

1. You get the current frame when Sen drains the inputs.
2. You compute the next frame based on the current one.
3. Sen prints the next frame when it commits the outputs.

During the drain stage

Sen will:

1. Update the time value (a value that you can fetch using the API).
2. Update the properties of the objects that have changed.
3. Call the methods that other components have called on your objects.
4. Perform the callbacks that have been triggered due to the emission of events.

During this stage you might update the state of an object or emit an event. That's fine. Those changes will not be visible to anyone (including you) until your outputs are committed.

During update stage

Sen will call the update() function on the objects that you have created. Sen will keep track of the properties that you have changed, the events that you have emitted and the methods that you have called.

You can fully rely on safely fetching the state of the objects that you are tracking. Even if they come from other components. They will not change. They have been created just for you. Even if the other component is running in the same process.

During commit stage

Sen will:

1. Collect all the changes that you made to your objects and transmit them to those who are interested. It does it in a batch, so that when others fetch your state, they will see a fully consistent (atomic) change.
2. Transmit the events that you produced to those who are interested in them.
3. Request the other components to perform a method call on the objects and methods that you called during the update stage.

Components do not run in any order relative to one another. Each has its own thread and its own freqHz, and nothing schedules one after another. The double buffer is what makes that safe: what you read during your update was fixed before it began. A component that has to act on another one's result therefore reads what that component last committed, never a value being computed in the same cycle, and you express such a dependency in the data flow instead of in the schedule. group orders startup and shutdown only, as Configuration says.

Different rates need nothing special. A slow reader sees whatever the fast component committed most recently, and a fast reader sees the same value on every cycle until the slow one commits again. Under virtualTime that does not change: freqHz still decides how often a component updates, and how far time moves on each step is the distance to the next component due to run.

Real-time execution vs stepped execution¶

"Real time execution" in Sen means that Sen will use the internal system clock to measure the passage of time. When you have multiple components running at the same time, you need to decide the mechanism you want to use to keep them in sync.

In case of real time execution, the usual approach, and the one that works best, is to have the computers' clocks in sync using PTP (or NTP), and select an update rate where every component updates frequently enough so that the execution progresses consistently forward within some margin of tolerance. Be aware that this approach is inherently non-deterministic, because the iterations are not fully coordinated and very much affected by the precision of the time sync, the compute load and scheduling made by the OS, the network load and transport delays, and a long list besides.

Stepped execution does coordinate your components. This idea translates to:

• Components do not advance with the natural passage of time. In fact, they do not advance until requested.
• The time that components fetch comes from an internal variable held by the kernel.

You ask for it in the configuration, with the kernel's runMode:

Under virtualTime the kernel publishes a clock object you drive, named clock by default and placed on the kernel's bus unless clockBus says otherwise. You advance it with processNoFlush(delta) and flushOutputs(), and keeping them separate is what makes several processes tractable: processNoFlush(delta) updates the time, drains the inputs and cycles the components without making their outputs visible, and flushOutputs() then publishes them. Across several processes you call the first on every kernel, wait for all of them, and only then call the second, which is how the whole system steps together instead of drifting apart. Setting clockMaster: true publishes a master clock that discovers the individual kernel clocks on its bus and does that coordination for you.

Within a single process, stepped execution is deterministic. Across processes it is not, yet.

Not everything can be stepped. A component that returns true from isRealTimeOnly() is left out of the set the kernel advances: it keeps following the real clock while the rest of the system is virtualized.

That is for components doing infrastructural work instead of simulating anything, the ones whose job only makes sense in real time. A shell stepped along with the model would be unusable: it would respond only when you advanced the clock, and you advance the clock from the term. The same reasoning covers transports, profilers and anything driven by a person or an external system.

Some of the shipped components are marked this way, for the reasons above, and keep following the real clock when the rest of the system is stepped.

Everything the kernel builds from your build: section is stepped. See Writing a component for how to mark one of your own.

The time a component sees¶

getTime() gives you the execution time, and what that time means is yours to decide: Sen moves it along and does not interpret it. Under realTime it follows your component's schedule, anchored when the component starts and advancing by exactly one period per cycle. Under virtualTime it advances by that same period, aligned to multiples of it, and no cycle is ever skipped. What changes is the stepping: you drive it, so how fast it runs against the wall clock is your choice and need not match it at all. It is fixed for the cycle you are in, so reading it more than once during update() gives the same value. If your system already keeps other clocks, you can correlate them to this one and drive them however you like.

A component that runs its own cycle can publish its data with a time of its own instead of Sen's. That is for bridging a system that keeps its own clock, and it is covered in Publishing with the time the data came from.

You drive that stepping from the master clock. step() takes a single step and steps(n) takes several, while advanceTime(duration) takes as many steps as fit in the duration and rounds up to a whole one, so asking for 100 ms where a step is 30 ms runs four steps and advances 120 ms.

The step size is not a setting. Each kernel clock reports how long until the next component it hosts is due, and the master takes the smallest, so time stops at the next thing that has to run. Steps therefore vary in length, and a component's freqHz still decides how often it updates. Controlling the clock has the configuration and a worked run.

Fixed-rate cycles are what Sen schedules today, and other scheduling modes will follow as they are needed.

Sen is not a simulation framework. What it gives you is objects other processes can see, a cycle that runs them and a clock you can drive. That is what a simulation framework would be built on, and Sen stops there. Solvers, scenario handling, model libraries and scheduling policies are yours to bring or to build. For a component that does not need any of that, what is here may be enough on its own.

A component that advances in time takes its step from the clock rather than from its configured rate. Keep the last time you saw:

  /// Empty until the first update, which has nothing to difference against yet.
  std::optional<sen::TimeStamp> lastUpdate_;

and difference it on every cycle:

    // Elapsed time comes from the clock, not from the configured period: a skipped cycle covers two
    // periods, and a component that is stepped rather than cycled has no period at all.
    // The first cycle has nothing to difference against, so it contributes nothing rather than the
    // whole interval since the kernel started.
    const auto now = runApi.getTime();
    const auto elapsedTime = now - lastUpdate_.value_or(now);

    // Advanced on every cycle, including while the timer is off, so switching it on does not
    // subtract the whole interval it spent paused.
    lastUpdate_ = now;

Subtracting two TimeStamp values gives a Duration. A component integrating a rate turns that into a float64_t with toSeconds() and multiplies by it; the timer subtracts it from a countdown. Because the step comes from the clock instead of from freqHz, the same code is correct when a cycle is skipped and when the component is stepped. It is not the same run, though. Stepping gives the same sequence of steps every time, while under realTime a skipped cycle merges two steps into one, so a component with a saturation, a rate limit or a discrete event can land somewhere else.

When a component runs out of time¶

Only under real-time execution. Stepping has no deadline to miss: the kernel waits for every component to finish its cycle, so a slow update() makes the run take longer without losing a cycle. Components marked isRealTimeOnly() keep their deadline even in a stepped system.

A component at freqHz: 30 gets 33 ms. If update() takes longer, the kernel does not stretch the period or queue the work: it skips whole cycles and stays on the original schedule. A component that overruns once loses updates but is back in step immediately, with no delay accumulating. The execution time follows that schedule, so it advances by a whole number of periods: two consecutive update() calls can be 33 ms apart or 66, but never 41.

Cycles are skipped for one other reason. Under real-time execution the schedule is kept on the machine's clock, so an adjustment to it, from NTP or PTP or by hand, moves the schedule too. A large correction forward skips cycles the same way an overrun does. A large one backward keeps the component running at its rate, but leaves its schedule ahead of wall time by the size of the correction, and nothing brings the two back together: closing the gap would mean either running the component below its rate for a while or letting it answer late, and neither is a trade Sen makes for you. If the high-resolution clock on your machine is the system clock, a correction arriving while the component is asleep holds it there for the length of the jump. Small corrections, which is what a synchronized network produces, pass through without any of this.

Overruns and missed frames are reported separately, and they are not the same:

An overrun is counted in CPU time, so an update that blocks on a socket, a lock or a vendor SDK burns wall time without burning CPU and never counts as one. The missed-frame lines are the ones that say cycles were lost, which is what a component running at a fraction of its configured rate produces.

The two missed-frame lines point in different directions. An interruption means the work finished after the cycle it belonged to, which can be because the update ran long or because the cycle started late. An oversleep means the sleep returned more than a full period late, so the component was not running at all when it should have been, which makes it a symptom of the machine and not of your code. They also differ in what happens next: after an oversleep, if less than half a period remains, the kernel skips the frame outright and waits for the next one instead of starting a cycle it cannot finish.

A run of lost cycles reports once, not once each. The kernel takes every missed cycle in a single pass and warns one time, so a component that blocks for five seconds at 30 Hz produces one line instead of a hundred and fifty. The counters below are not summarised that way: they count every cycle that was lost, so the same run adds all of them to missedFrameCount.

The warning can be suppressed from code but not from configuration: RunApi::execLoop takes a logOverruns flag, so a component driving its own loop can drop the log line and keep the Tracy message. Components declared under build: run through the kernel's standard pipeline, which does not take the flag.

Reading the numbers from your own code¶

RunApi::fetchComponentMonitoringInfo() returns what the kernel measured for the calling component. RunApi::fetchMonitoringInfo() returns the same for every component the kernel loaded.

The numbers are easy to confuse. This is what each one holds. The counters start at zero when the component starts running, never go down, and are never reset.

Some of your code is not counted in lastCycleComponentCpuTime: preDrain(), preCommit(), the handlers that run when your outputs are committed, and the callbacks for a source appearing or disappearing. Those go with Sen's time.

CPU time is not the same as elapsed time. A cycle that waits on a socket, a lock or a library uses wall time but no CPU. It does not raise overrunCount, and it raises missedFrameCount once it runs past its cycle. So if lastCycleExecutionCpuTime stays well below the period and frames are still missing, the component was waiting, not computing, and looking for slow code will not find anything.

An overrun means the work cost too much for the period. A missed frame means a cycle did not run. A cycle that overruns always misses a frame as well, because it cannot use more CPU time than the time that passed. The other way round is more common: a component that waits misses frames without overrunning.

Check lastCycleStartDelay before you blame your own code. If the thread wakes late by less than a period, that is not counted as an oversleep. The cycle still starts late, finishes past its slot, and the cycle it loses goes to missedFrameCount. So frames going missing while the start delay is large means the machine was late, not that your work is too slow. A clock correction that moves the schedule forward does the same thing.

oversleptCount counts cycles where the thread was not running when it should have been. That is the operating system, not your code.

The counters do not overlap. overrunCount counts cycles that ran and cost too much. The missed-frame counters count cycles that never ran, and never the same cycle twice, so missedFrameCount + oversleptCount is how many cycles were lost.

These numbers are only filled in when the kernel runs the cycle. Under virtual time there is no deadline to miss, and a component that drives its own loop has no schedule of ours, so in both cases you get nothing rather than zero. A component marked isRealTimeOnly() keeps its deadline in a stepped system, and keeps these numbers with it.

The first cycle is left out of missedFrameCount and of the worst cycle. The schedule starts before the component has finished starting up, so that cycle is always late, and it carries whatever queued up while it was starting. If we counted it, every component would begin with lost cycles it never lost, and that one cycle would hold the worst-cycle mark for the rest of the run. An overrun or an oversleep on it is still counted, because both of those are real.

If a component misses one cycle in a thousand, the cycle you read is almost never the bad one. The last-cycle numbers tell you what is normal, and the counters tell you something went wrong. The worst cycle is the one that used the most CPU: how much it used, and how much of that was yours. It is not the slowest cycle. A cycle that waits uses almost no CPU, so it never becomes the worst one, and you find that one through the counters and the start delay.

The numbers do not all come from the same cycle. Each one is stored as the cycles run, so a single reading can mix a last cycle with a counter that has moved on. The worst-cycle pair is the exception: it is always one cycle, so the share can be compared with its total. For a rate, read twice and subtract.

When a cycle runs long, this is where to look. If most of the time is in your objects' update() and your work function, that is your own code. If most of it is in the queued work, the cost is in the calls other components make on you and in the callbacks you registered. If what is left over is the largest part, the time goes to the bus, draining what arrived and committing what changed, which grows with how much your component subscribes to and publishes. And if a cycle takes a long time while its CPU time stays small, it was waiting. None of these numbers show that, so time it yourself inside update().

On Windows these numbers are coarse. Thread CPU time comes from the system clock tick, which is 15.6 ms by default. For a component faster than about 60 Hz, the CPU numbers and the overruns counted from them say very little.

Note: With stepped execution the system is not only deterministic within a process, but multithreaded. If components have a significant amount of work to do on each iteration, and there are multiple cores (or computers), the usage of your compute resources improves. This is a nice (and non-accidental) side effect of the drain-update-commit approach used with threads and processes.

Also Note: Having multiple processes implies a higher synchronization overhead. Data flows have to be synchronized not only between threads, but between computers. If the work that components do is outweighed by the synchronization overhead, the result is a pessimization. Pessimizations are common when trying to parallelize, as many factors come into play. Luckily, Sen allows you to compose your system as you need without touching a single line of code, so you will have an easy time shaping your configuration according to the performance profile of your computations.