The ether transport¶

If you load the ether component, your process will be able to communicate with other processes.

load:
  - name: ether
    group: 1

Sen processes discover each other using multicast. Normally, you don't need to change the default discovery parameters, but here you can see the available options:

package sen.components.ether;

optional<string> MaybeDeviceName;

// TCP-based discovery hub parameters
struct DiscoveryHubAddress
{
  port : u16,     // Port where the discovery hub is running
  host : string,  // name or IP of the host where the discovery hub is running
}

// TCP-based discovery parameters
struct TcpDiscovery
{
  beamPeriod             : Duration,             // How often are we beaming our presence
  beamExpirationTime     : Duration,             // Expiry time used by the beam tracker
  hubAddress             : DiscoveryHubAddress,  // Where to find the discovery hub
  allowVirtualInterfaces : bool                  // If true, virtual (NO_CARRIER) interfaces are also considered
}

// Multicast-based discovery parameters
struct MulticastDiscovery
{
  beamPeriod             : Duration,         // How often are we beaming our presence
  beamExpirationTime     : Duration,         // Expiry time used by the beam tracker
  mcastGroup             : string,           // Defaults to '239.255.0.44'
  port                   : u16,              // Defaults to 60543
  device                 : MaybeDeviceName,  // Device to use for the network communication
  allowVirtualInterfaces : bool              // If true, virtual (NO_CARRIER) interfaces are also considered
}

// Configuration parameters related to the discovery protocol
variant DiscoveryConfig
{
  MulticastDiscovery,
  TcpDiscovery,
}

// What to do when a queue is full
enum QueueEvictionPolicy: u32
{
  dropOldest,  // When full, drop the least recent element
  dropNewest,  // When full, drop the most recent element
}

struct QueueConfig
{
  evictionPolicy : QueueEvictionPolicy,  // What to do when a queue is full
  maxSize        : u64,                  // Maximum element count, 0 means unbounded
  warningLevel   : u64                   // Element count where warning will be issued, 0 means none
}

// Range of values of a u8
struct ByteRange
{
  min : u8,  // minimum range
  max : u8   // needs to be >= min
}

// Range of values of a IPv4 multicast address
array<ByteRange, 4> MulticastRange;

// Inclusive IPv4 multicast address range
struct MulticastAddressRange
{
  min : string,
  max : string
}

sequence<MulticastAddressRange> MulticastAddressExclusions;

// Inclusive port range
struct PortRange
{
  min : u16,
  max : u16
}

sequence<PortRange> PortExclusions;

// Let the operating system select an ephemeral port
struct Ephemeral {}

// Bind exactly this port, fail closed if taken
struct PinnedPort
{
  port : u16
}

// Bind within [min,max], random start, walk forward
struct ProbePortRange
{
  min : u16,
  max : u16
}

variant PortBinding
{
  Ephemeral,
  PinnedPort,
  ProbePortRange
}

// This configuration is per process and does not need to match across Sen instances.
// Exclusions apply to a pinned port, not to an ephemeral one: an ephemeral bind asks the OS for
// any free port and does not learn which it was given.
// udpUnicast cannot be pinned. Its socket is per peer and UDP has no four-tuple to demultiplex
// on, so a second peer either fails to bind or silently takes the first peer's traffic.
struct PortConfig
{
  tcpAcceptor : PortBinding,  // inbound TCP listener
  udpUnicast  : PortBinding,  // per-peer UDP unicast; must be Ephemeral
  tcpSource   : PortBinding   // outbound TCP source port
}

optional<PortConfig> MaybePortConfig;

// Remember to set these to the same value in all related Sen instances
struct BusConfig
{
  // Defaults to the whole of 239.0.0.0/8: [239], [0-255], [0-255], [0-255].
  // RFC 2365 assigns only 239.192.0.0/14, which holds 262144 addresses and is not enough to keep
  // many bus names apart. Set [239], [192-195], [0-255], [0-255] to stay inside it.
  // Remember to keep a range wide enough for avoiding collisions if you have many buses.
  multicastRange : MulticastRange,

  // Address ranges that must not be allocated
  multicastExclusions : MulticastAddressExclusions,

  // Defaults to 50985
  // Change this in case you have limited control over the infrastructure.
  multicastPort : u16,

  // If true, no multicast traffic is performed and all the bus settings above
  // become meaningless. Defaults to false.
  multicastDisabled : bool
}

optional<u16> MaybeDiscoveryHubPort;

// Overall component configuration
struct Configuration
{
  discovery          : DiscoveryConfig,        // Discovery-related parameters
  busOutQueue        : QueueConfig,            // Queue for broadcasting to buses
  processUdpOutQueue : QueueConfig,            // Queue for communicating with processes over UDP
  processTcpOutQueue : QueueConfig,            // Queue for communicating with processes over TCP
  busConfig          : BusConfig,              // Bus transport configuration
  portExclusions     : PortExclusions,         // Port ranges that this process must not use
  portConfig         : MaybePortConfig,        // If not present, all port bindings are Ephemeral
  runDiscoveryHub    : MaybeDiscoveryHubPort,  // If present, run the TCP discovery hub on this port
  networkDevice      : MaybeDeviceName         // If present, the network device to use
}

If you arrive from another distributed system, the word you are reaching for here is peer. Sen calls it a remote kernel: the kernel in another process that this one has found, one per process. That is the term used in this documentation. Peer is kept only for the shape of the network: Sen's transport is peer-to-peer, with no central server or broker to connect to.

Queue sizes¶

If the components are pumping more data than the I/O stack can handle, your system will eventually run out of memory. So cap the queues, and watch the warnings warningLevel raises under load.

Isolating communication¶

If you are sharing a network with other Sen instances that are unrelated to your project, you can set the discovery port to a value shared only by the Sen instances you want to keep together. discovery is a variant, so it takes a type and a value:

load:
  - name: ether
    discovery:
      type: MulticastDiscovery
      value:
        port: 60544

A flat discovery.port: is not an error and is not applied. The parser needs the type key to know which alternative you mean, and without it the whole block is ignored and the default port stands.

If changing the YAML file is not possible or convenient, this parameter can be overwritten by using the SEN_ETHER_DISCOVERY_PORT environment variable.

Working with containers¶

Sen uses the host name to identify participants in the network. If running a container, remember to set a host name so that Sen gets the right picture about where things are running (you can do it with the --hostname option).

If you are using the TCP Discovery mechanism in a bridged network, remember to set hubAddress.host in the discovery configuration to the host name or IP address of the container hosting the hub.

Network interfaces¶

The ether component needs a working network interface enabled with IPv4, and multicast. If you don't have any network interface, please ensure that you have the loop-back interface enabled and with multicast support.

Ensuring that you are using valid interfaces¶

If there are no network interfaces, the ether component will not start.

By default, the ether component ignores virtual interfaces (interfaces that have the NO_CARRIER flag). This is useful when working with containers, but might be inconvenient when working with virtual machines.

You can enable virtual interfaces with allowVirtualInterfaces, nested inside the discovery variant as above, or by setting the SEN_ETHER_ALLOW_VIRTUAL_INTERFACES environment variable to true (or yes, or a number != 0). The environment variable is the simpler of the two.

Ensuring that multicast is enabled¶

sudo ip link set lo multicast on
sudo ip route add 224.0.0.0/4 dev lo
sudo ip route add ff00::/8 dev lo

If you have other interfaces, use eth0 (or your preferred interface) instead of lo.

Remember to also do it if you are inside a Docker container (use --cap-add=NET_ADMIN and eth0).

Configuring process ports¶

By default, Sen asks the operating system to select the TCP and UDP ports. The optional portConfig parameter has one independent entry for each process port, and each entry can use any supported port binding mode:

• tcpAcceptor: TCP listening port used to accept incoming process connections.
• tcpSource: TCP source port used when initiating outgoing process connections.
• udpUnicast: UDP unicast port used for per-peer best-effort traffic.

Each entry accepts the same port binding modes:

• Ephemeral: let the operating system select the port. This is the default.
• PinnedPort: use one exact port.
• ProbePortRange: try ports inside a configured range until one can be bound.

When ProbePortRange is enabled, Sen starts from a random point in the range and scans forward with wrap-around and bind the first available port. It fails if no port in the range can be used. Pinned ports fail if the configured port cannot be used. This configuration is local to each process and does not need to match in other Sen applications. If portConfig is omitted, all process ports use Ephemeral.

load:
  - name: ether
    portConfig:
      tcpAcceptor:
        type: PinnedPort
        value:
          port: 20000
      udpUnicast:
        type: Ephemeral
      tcpSource:
        type: ProbePortRange
        value:
          min: 21000
          max: 25000

Excluding process ports¶

The portExclusions parameter declares port ranges that PinnedPort and ProbePortRange must not use. It does not affect to Ephemeral, where Sen asks the operating system to choose the port. Sen also avoids well-known ports (0-1023) and operating system ranges reported as dynamic/ephemeral.

load:
  - name: ether
    portExclusions:
      - min: 10000
        max: 20000

Controlling multicast¶

Sen uses multicast to distribute information to multiple receivers with minimum overhead. The multicast groups are generated based on an internal algorithm. In some cases, you might need to deploy Sen applications in a context where multicast support is limited.

Use the network footprint report to inspect the addresses and ports that each process configuration is expected to use.

That report is built from the configuration, before anything runs, so it shows what a process is expected to use. A running process can be asked directly, through its kernel object: KernelApi.getRuntimeNetworkFootprint() returns the buses it has open and the ports it has bound, including the ephemeral ones the operating system picks at startup. When multicast is disabled no multicast bus is created, and the report follows.

Setting the network interface¶

You can set the networkDevice attribute to force Sen to use a particular network interface. If set, the ether component will route all the traffic through it. To know which devices can be used, you can execute the ip a command (in Linux). Typically, the names will be along the lines of lo, eth0, or similar.

Take into account that:

• The interface must be UP.
• If you use multicast, multicast needs to be enabled.
• You may need to configure your OS IP routing.

Setting multicast ranges¶

The busConfig.multicastRange configuration parameter defines the ranges for possible multicast groups to be used by the buses. The default range is the whole of 239.0.0.0 to 239.255.255.255. In order for this to work, you need to ensure that all the related Sen applications are using the same range: a bus address is a hash of the bus name taken over the usable addresses, so two applications with different ranges or different exclusions derive different addresses for the same bus and never meet on it.

RFC 2365 assigns 239.192.0.0/14 for private use and asks that the rest of 239.0.0.0/8 be left unassigned until that is not enough. It is not enough here. The address is a hash, so collisions follow the birthday problem: within 239.192.0.0/14 two of 500 bus names share an address about 38% of the time, against under 1% over the whole block. Sen therefore allocates from all of it. If you would rather stay inside the assigned scope, set the range to 239.192.0.0 to 239.195.255.255 and keep the bus count well below the point where Sen starts warning about collisions.

Some of the block is never allocated:

Additional address ranges can be excluded with busConfig.multicastExclusions. Bear in mind that the exclusions take part in the derivation, so every application sharing a bus needs the same ones.

load:
  - name: ether
    busConfig:
      multicastExclusions:
        - min: 239.192.0.0
          max: 239.192.99.255
        - min: 239.192.200.0
          max: 239.192.255.255

Excluding ports¶

portExclusions lists inclusive port ranges this process must not use. Sen also asks the operating system which ports it has reserved and avoids those as well; where it cannot ask -- a container without /proc/sys/net/ipv4/ip_local_port_range, or Windows without WMI -- it logs a warning and continues with only the ranges configured here.

load:
  - name: ether
    portExclusions:
      - min: 8000
        max: 8080

Disabling multicast for bus traffic¶

You can force Sen to use TCP. The busConfig.multicastDisabled configuration parameter can be used for this purpose. You can also set the SEN_ETHER_DISABLE_BUS_MULTICAST environment variable with true (or yes, or a number != 0). TCP is resilient to message drops or re-ordering, but it does not scale in regard to the number of participants, as emitters will have to send dedicated messages to all receivers.

Changing the port for bus traffic¶

Bus traffic uses port 50985 by default. Every participant has to use the same one, so changing it moves an instance rather than the protocol: set it everywhere or they will not hear each other. The SEN_ETHER_BUS_MULTICAST_PORT environment variable overrides it, as SEN_ETHER_DISCOVERY_PORT does for discovery.

This matters most on Windows, where the Host Network Service reserves blocks of ports for itself even when nothing is listening on them. If a reserved block contains 50985, the bind fails with "an attempt was made to access a socket in a way forbidden by its access permissions" and Sen does not start. netsh int ipv4 show excludedportrange protocol=udp lists the reserved blocks. Pick a port below 49152, which is outside the range Windows reserves from.

Disabling multicast entirely¶

By default, Sen relies on multicast for kernels to discover each other. If your infrastructure does not allow multicast traffic at all, you will need to disable it for bus traffic (see previous section) and enable the TCP-based discovery hub.

The TCP discovery hub is a process that Sen applications connect to in order to discover each other. You only need one.

Tell the ether component to run one, and give it a port.

load:
  - name: ether
    runDiscoveryHub: 64222     # if you set this, we will run a discovery hub at this port

If you want to fully disable multicast you would need something like this:

load:
  - name: ether
    busConfig:
      multicastDisabled: true  # no multicast bus traffic
    runDiscoveryHub: 64222     # if you set this, we will run a discovery hub at this port
    discovery:
      type: TcpDiscovery
      value:
        hubAddress:
          host: localhost
          port: 64222           # has to match with the port where the hub will be running

Point the discovery configuration at it:

load:
  - name: ether
    busConfig:
      multicastDisabled: true  # no multicast bus traffic
    discovery:
      type: TcpDiscovery
      value:
        hubAddress:
          host: theComputer  # this can also be an IPv4 address
          port: 64222        # has to match with the port where the hub will be running

Configuring the discovery expiration time¶

Different ether instances find each other using a beamer that broadcasts beam messages periodically. The period at which beams are sent can be configured in the DiscoveryConfig of the ether configuration by modifying the beamPeriod. The default value of this period is 1 second.

The BeamTracker watches for those beams and uses beamExpirationTime to decide when one has stopped arriving at the expected frequency, at which point it is assumed lost. It defaults to three times beamPeriod, which is too tight once you set the period low, 100 ms say, so you can set it yourself in either of these ways:

• Configuring the beamExpirationTime parameter of the DiscoveryConfig to the desired duration.
• Setting the BEAM_TRACKER_EXPIRATION_TIME_MS environment variable to the desired duration in milliseconds.

Until a beam expires, the peer's objects are still there, holding the values from the last update that arrived, and nothing marks them stale. When it expires the peer is treated as gone and its objects are removed like any other departure, as Main concepts describes.

So beamExpirationTime is the window in which a model can read values from a machine that is no longer there. Shortening it narrows the window and makes a brief stall more likely to be read as a departure.

UDP OS buffer sizes¶

Some operating systems, most notably Linux, place restrictive limits on socket buffer sizes. Raise them to at least 8MB before pushing large amounts of UDP traffic through your instance. 8MB is a starting point rather than a ceiling, and can go higher.

Linux¶

Check the current UDP/IP receive buffer limit & default by typing the following commands:

sysctl net.core.rmem_max
sysctl net.core.rmem_default

If the values are less than 8388608 bytes you should add the following lines to the /etc/sysctl.conf file:

net.core.rmem_max=8388608
net.core.rmem_default=8388608

Changes to /etc/sysctl.conf do not take effect until reboot. To update the values immediately, type the following commands as root:

sysctl -w net.core.rmem_max=8388608
sysctl -w net.core.rmem_default=8388608