Skip to end of metadata
Go to start of metadata

Flux Agents

Agents are lightweight Flux add-ons that allow the remote execution of any of the following tasks:

  • Process Action (invoke command-line programs and scripts)
  • File Copies
  • File Moves
  • File Mods (from 8.0.11)
  • File Polling (trigger a workflow when a file appears or when a file disappears)

Note that agents do not run workflows. Agents instead run actions and triggers within the context of a workflow that is running on a Flux engine or a cluster of Flux engines.

Agents are not a replacement for the Flux engine – rather, they augment the functionality of the engine by allowing it to send these specified tasks to be executed remotely on the network. For this reason, at least one engine is necessary in order to use Flux agents.

Some of the major benefits of agents:

  • Lightweight: Agents have a lighter footprint than engines and are ideal for machines where only a small subset of the overall process must be run.
  • Scalable: Agents and engines communicate efficiently and can be spread across a wide network.
  • Remote Execution: Agents run tasks remotely from the Flux engine, letting you access your processes wherever they may need to be run.
  • Targeted Execution: Tasks can be targeted to specific agent pools, ensuring your tasks only run on the machines with the necessary resources.
  • Reporting: Agents send their results back to Flux, allowing them to be handled and reported using the traditional Flux mechanisms.
  • DMZ and Firewall Friendly: Agents do not need to open any incoming ports, allowing them to safely receive work behind a firewall or inside your DMZ.

Flux Engine-Agent relationship

The Flux engine (or cluster) coordinates all tasks that are to be executed on the remote agents. Engine-agent communication has two simple requirements:

  • The Flux engine's machine must allow incoming traffic on the engine port (by default, TCP/IP port 7520).
  • The Flux agent must allow outgoing traffic on the same port.

That's it! As long as these requirements are met, the Flux engine and agent can communicate. You do not need to open any incoming ports on the agent side.



List of Configuration Properties

The following table contains a list of configuration properties that can be set in your agent configuration, along with a description of each, and the default value (if any) that is used if you do not set the configuration property yourself.

 

 

Property
Description
Default Value

CONCURRENCY_THROTTLE 

The number of processes that can execute simultaneously on this agent. 

10

ENGINE_ENCRYPTED_PASSWORD 

The encrypted password that the Agent uses to log in to a remote Flux engine. Encrypted passwords can be generated using the Engine's command line interface. This property can only be used if ENGINE_PASSWORD is null or empty. 

none (this property is not set by default) 

ENGINE_HOST 

The host name of the Flux engine with which an agent intends to register. When an agent registers with an engine, the agent effectively registers with all engines in that cluster. This property must be set to allow the agent to look up the engine. 

none (this property is not set by default and must be explicitly configured) 

ENGINE_PASSWORD 

The password this agent uses to login to a remote Flux engine. This property can only be set if ENGINE_ENCRYPTED_PASSWORD is null or empty. 

none (this property is not set by default) 

ENGINE_PORT 

The TCP/IP port that the remote Flux engine is bound to. The agent will use this port to look up the remote engine. 

7520

ENGINE_USERNAME 

The username that this agent will use to log in to the remote Flux engine. 

none (this property is not set by default) 

ID_DESCRIPTION 

A description of the agent instance intended for human consumption. 

none 

ID_FULL_NAME 

A long, descriptive name of the agent instance, intended for display purposes. 

none 

ID_NAME 

The case-insensitive name of the agent instance. Used to differentiate agent instances in the logs, the audit trail, and Flux security login modules. If the name is not set explicitly in a configuration, the configuration will generate a name using:

  1. The host name of the local computer but not "localhost". For example, "foo" or "foo.bar.com".

  2. Failing that, the IP address of the local computer but not "127.0.0.1". For example, "12.34.56.78".

  3. Failing that, "unknown".
    If there is more than one active agent in a JVM, each generated name is made unique. To make a generated name unique, a ":" symbol is first appended to the name that is generated as described above. A number, starting from one, is then appended to ensure that the generated name is unique within the JVM. 

    In Flux 8.0 agent ports are no longer needed; you only need to setup the name of the agent, and point it to an engine and the engine port.

determined on agent startup (see description column) 

INTERNAL_LOGGER_FILE_DIRECTORYThe directory where log files are placed when using the internal logger.. (equivalent to the system property "user.dir", typically the directory where the engine or JVM was started)
LOG_FILENAMEThe path (relative or absolute) and filename of the log file that this agent should use for its logging data.

Relative paths are translated relative to the directory where Flux was started – usually, the Flux installation directory.
flux-agent.log

POOL 

The (case-insensitive) name of this pool that the agent will belong. Agent pools allow you to dictate where Process Actions send their work to be executed – a Process Action may run its work on any available agent, or it can use a pool to specify a specific agent or group of agents to execute the work. If an agent's POOL configuration setting matches the value that a Process Action uses, that agent will be eligible to run the work for the Process Action.

This property can be set to any value except '*' is a special character used in Process Actions to indicate that the work can be run on any available agent, regardless of the pool. 

none (agents do not belong to any pool by default) 


As with engine configurations, agent configurations use the standard Java properties file format <Property name>=<value>. A sample agent configuration file might look like: 

 

Agent Configuration

NOTE: Agents require that the Flux engine has CLUSTER_NETWORKING_ENABLED=true

Creating a Flux Agent

For your consideration, Flux includes a script for launching a Flux agent. This script, start-agent, is included in the installation directory of your Flux package.

If you installed Flux using the Windows installer, an Agent is automatically installed as a Windows service as well.

If you run your agent using the default script or service, you can find its configuration settings in config-agent-config.properties. If you're running Flux on a remote machine for the first time, you'll probably need to specify your ENGINE_HOST and ENGINE_PORT to match the name and location of the Flux agent you want this agent to receive instructions from.

Agents can also be created and started from the Command Line Interface, like so: 

Creating a Flux Agent from the command line

Where path/to/agent-config.properties is the path to an agent configuration file. The configuration properties supplied in the configuration will determine how the agent is created and how it connects to the Flux engine.

You can also specify configuration settings directly on the command line. When specifying a configuration setting, the name of the property should be applied in lowercase, with all '_' characters removed, followed by a space, then the value of the configuration property. For example: 

Start a Flux Agent from the command line with specific options

States of a Flux Agent

The diagram below outlines the various states of the Flux agent:

Once an agent is created, it enters its initial state, registering. From this state, it will either register successfully and move to the STOPPED state, or it will continue trying every 3 minutes until it is able to successfully register.

From the STOPPED state, the agent can be started, at which point it is FREE. A FREE agent can accept work from a Flux engine.

Once some work arrives from the engine, the agent is BUSY. A BUSY agent is an agent that is actively executing a task. If an agent is BUSY and its CONCURRENCY_THROTTLE is full, the agent will not accept any new work until at least one existing item is finished.

From any state, the agent can be DISPOSED, where it will stop executing running until it is started again, where it will repeat this cycle.


  • No labels