The engine configuration defines both how your engine runs, and how it stores and maintains data from the Flux engine (including workflow and user data).
The engine configuration settings can be configured using the Operations Console, in a .properties file, or, if you are starting the engine using Java code, a flux.Configuration object. These configuration settings cannot be changed after the engine has started. For a comprehensive list of static configuration options, see the Javadoc for the flux.Configuration option (Javadocs are located in the /doc/javadoc directory under your Flux installation directory).
Engine behavior is also controlled by the Runtime Configuration and Runtime Variables. The runtime configuration and the engine configuration are used together to define how the engine operates.
List of Configuration Properties
The following table contains a list of configuration properties that can be set in your engine configuration, along with a description of each, and the default value (if any) that is used if you do not set the configuration property yourself.
A time expression that specifies the duration of time between agent connectivity tests. If an agent is busy and cannot be reached during this time period, the process assigned to the unreachable agent will be reassigned to the next available agent in the pool.
A time expression indicating how old individual audit trail, run average, and run history entries must be before they are automatically pruned. Expired entries are automatically deleted from the database. A null audit trail expiration indicates that these entries are never deleted from the database.
+1H (when using the H2 in-memory database), OR
A list of audit trail events to record to the audit trail. If this audit trail filter is not set or empty, all audit trail events are recorded to the audit trail.
none (by default, all audit trail events are recorded)
The name of the logger used when logging audit trail information.
Indicates whether actions that execute in a workflow that are later rolled back in the curent database transaction are still recorded to the audit trail. Useful for knowing what actions led up to an error, before the error caused the current database transaction to be rolled back.
The initial size of the workflow cache. The cache size determines the number of workflows that can be stored in the cache.
Flux will dynamically increase the size of the cache as needed to accommodate larger loads of workflows that arrive.
This configuration property controls how workflows are cached.
Depending upon the nature of your workflows, there are instances where LOCAL cache performs more slowly than NONE. This is especially true with very large workflows involving many actions and variables.
The name of the logger used when logging client-side information.
Indicates whether networking is enabled in the engine instance when it participates in a cluster. Networking is not required for clustering, but networking increases responsiveness among engine instances across the cluster. By disabling cluster networking, the engine instance still participates in a cluster, but all its communication with other engine instances is performed through the database server.
The file transfer monitor user interface and the execution of Flux agents depend on cluster networking being enabled. These features will be disabled when cluster networking is turned off.
Note that it does take longer to remove workflows from the engine if cluster_networking is enabled. A "You are attempting to modify workflow ..." error may occur if workflows are being added to the engine, immediately executed, and then resubmitted using the same names. In such instances it is recommended that cluster_networking be disabled.
The concurrency throttle limit in the root node of the runtime configuration tree. This limit controls the number of flow contexts that may execute at one time across the entire engine.
This property is only used if a runtime configuration file is not being used. If a runtime configuration file is present, the Flux engine will use the CONCURRENCY_THROTTLE specified in the runtime configuration, or a default of 10 if none is specified.
A time expression specifying the time at which the engine performs once-a-day maintenance on the Flux database tables. For example, to run daily maintenance each morning at 04:30, set this configuration property to the Cron-style time expression "0 0 30 4". The time zone is always the local time zone.
When choosing the time expression used, make sure maintenance is only performed exactly once per day. Do not run maintenance multiple times a day as this may interfere with Flux processing.
0 0 0 2 (By default, maintenance is done at 02:00 each morning)
The JNDI name used to lookup an application server's database connection pool.
none (you must explicitly set this property in order to use a data source)
Indicates whether to disable auto-commit on database connections obtained through a data source. If this property is enabled, the engine will check every connection retrieved from the data source for an auto-commit value of "false". If the connection has an auto-commit value of "true", the engine will log a warning and automatically attempt to disable auto-commit on the connection.
Indicates whether data source objects are saved and re-used later. Such caching of data source objects reduces warning messages emitted by WebSphere 5.0. Unfortunately, this caching may also reduce the ability of an application server to perform clustering. If possible, it is preferred to keep this configuration property at its default setting of false, which means that data source objects are not cached.
The password used to retrieve a database connection from an application server data source. This property is only used if it is not null and not the empty string, the data source username is not null and is not the empty string, and a data source has been set.
"" (empty string)
The JNDI name of a javax.transaction.UserTransaction object that should be used for client calls into the Engine. User transactions are used to guarantee transaction integrity when the engine's transactions need to be integrated with transactions from other software systems.
The JNDI name of a javax.transaction.UserTransaction object that should be used for workflows and background tasks running on the engine. User transactions are used to guarantee transaction integrity when the engine's transactions need to be integrated with transactions from other software systems.
A time expression specifying the timeout for all user transactions that the engine uses. If this property is null, Flux will not set timeouts for user transactions.
+d (one day)
Indicates whether a User Transaction object manages database transactions that are retrieved from a data source. If true, a javax.transaction.UserTransaction object is used to manage transactions. If false, Flux manages transactions.
The username used to retrieve a database connection from an application server data source. This property is only used if it is not null and not the empty string and a data source has been set.
"" (empty string)
The full path to a database properties file. Database properties allow you to specify how Flux should connect to the database, to rename database tables and columns, to reassign a standard SQL type, and to provide initialization SQL statements to database connections that are created to directly access the database.
The type of database that the engine should use.
The class name of a JDBC driver. If DATABASE_TYPE is DERBY or H2, this property is ignored (as Flux automatically manages drivers for those database types).
A time expression that specifies how often engine contents status audit trail events are generated. The engine contents summary audit trail event provides information on all of the workflows, messages, and message publishers that exist on the engine. Although this property allows you to adjust the frequency of these events, it is not possible to disable the engine contents summary completely.
+1d (one day)
A time expression that specifies how often the engine checks to see whether other engines in the cluster have failed. This is used for failover purposes to determine whether any workflows have prematurely stopped running and must be restarted on another engine.
Note that if an engine goes down and comes back up, the number of reported firing workflows can be up to double what would normally be expected of that recovering engine. This is because the recovering engine, upon startup, will schedule and begin firing a number of workflows equal to its concurrency level. But the engine will have already marked workflows as firing before it went down. These workflows will not be recovered, and their firing state changed, until the FAILOVER_TIME_WINDOW has been exceeded. Once the FAILOVER_TIME_WINDOW has passed the amount of concurrently firing workflows will return to the expected concurrency level.
+3m (three minutes)
Ensure that all servers in the Flux cluster have the same time value set. Use a network time service to synch times between Flux servers. To not do so can cause Flux actions to fire multiple times if the time drifts between servers.
A time expression that specifies how often Flux should temporarily increase workflow priorities to prevent lower-priority workflows from being starved (not allowed to run) by higher-priority workflows. A non-running workflow's priority will be temporary updated each time Flux performs the fairness check. The workflow's priority may be temporarily increased several times before it is able to run. Once the workflow does run, its priority will be reset to its original value and it will begin waiting again for the next opportunity to execute.
none (fairness checks are not performed by default – setting a value for this property will automatically enable them)
Indicates whether debugging is enabled for file transfers. If this property is enabled, all interaction between Flux and any remote file hosts will be logged to the Flux system log. When enabled, this property requires the INTERNAL_LOGGER_LEVEL to be set at FINE logging level or greater.
Indicates whether workflow deadlines (SLAs or Service Level Agreements) are enabled; disabling deadlines reduces database activity. When this property is enabled, you can set deadlines on individual workflows. If a workflow exceeds its deadline, it will publish an audit trail event indicating that the deadline has passed. The Operations Console will also graphically indicate any workflows that are approaching (or have exceeded) their deadlines.
The name of the logger used when logging information during workflow execution.
A time expression that specifies the frequency with which heartbeat audit trail events are generated. A heartbeat audit trail event provides information on the health of the engine. Although this property allows you to adjust the frequency of these events, it is not possible to disable heartbeats completely.
+1H (one hour)
The host name or IP address that the engine should bind to. If there are multiple host names / IP addresses available on the server where the engine is running, this property allows you to specify which one Flux should use for network requests.
If you are using this configuration to look up an existing engine, this property allows you to specify the host name where the engine is running.
|Flux will automatically select a host name from the available names on the system|
A description of the engine instance intended for human consumption.
The case-insensitive name of the engine instance. Used to differentiate Flux engine instances in the logs and audit trail. If the name is not set explicitly in a configuration, the configuration will generate a name using:
determined on engine startup (see description column)
The JNDI name of an application server's initial context factory.
The password used to create an application server initial context when looking up objects in an application server's JNDI tree (like data source, EJB, or JMS information). This property is only used if it is not null and not the empty string and the INITIAL_CONTEXT_USERNAME configuration property is not null and is not the empty string.
"" (empty string)
The username used to create an application server initial context when looking up objects in an application server's JNDI tree (like data source, EJB, or JMS information). This property is only used if it is not null and not the empty string.
"" (empty string)
The directory where log files are placed when using the internal logger.
. (equivalent to the system property "user.dir", typically the directory where the engine or JVM was started)
The maximum size to which a log file for the internal asynchronous and internal synchronous loggers can grow before it is rotated. At most, there are seven log files, which include the current log file and up to six previously rotated log files. The rotation size is specified in bytes.
10485760 (ten megabytes, 10 * 1024 * 1024)
The logging level used for the internal asynchronous and internal synchronous loggers. This property does not affect the logging level for any other loggers.
When Flux is configured to obtain database connections directly through JDBC (rather than a data source), it creates a connection pool and periodically recycles connections in the pool. This property is a time expression that indicates how often Flux should recycle database connections.
+1H (one hour)
The encrypted password used to obtain database connections when Flux is configured to obtain database connections directly through JDBC (rather than a data source). This property is only used if it is not null and not the empty string, the JDBC_USERNAME configuration property is not null and is not the empty string, and the JDBC_PASSWORD configuration property is null or the empty string.
"" (empty string)
The password used to obtain database connections when Flux is configured to obtain database connections directly through JDBC (rather than a data source). This property is only used if it is not null and not the empty string and the JDBC_USERNAME configuration property is not null and is not the empty string.
The username used to obtain database connections when Flux is configured to obtain database connections directly through JDBC (rather than a data source). This property is only used if it is not null and not the empty string.
(Available in Flux 8.0.13)
Keeps completed workflows on the engine and in the Flux Ready Table. Useful for development and testing of workflows. Note that if running thousands of workflows to completion per day, this option can cause performance slowdowns if the completed workflows are not periodically removed.
A time expression indicating how old log entries in the database must be before they are pruned (deleted) from the database. At the DAILY_MAINTENANCE_TIME each day, Flux will find and prune all entries older than this time expression.
+w (one week). If the DATABASE_TYPE is H2, however, the default value is ?+1H (one hour).
Indicates which logging system Flux should use. This property should only be set when a single logging system is used on the engine; for multiple loggers, use the LOGGER_TYPES property instead.
Indicates which logging systems Flux should use. This property should only be set when multiple logging systems are to be used on the engine (for example, this could be configured to use both log4j and the internal asynchronous logger to log messages).
The maximum number of database connections that the engine should be allowed to have open at once.
By default, Flux uses standard JDBC APIs for persisting large binary data and characters to the database. Due to an Oracle limitation, however, standard JDBC cannot persist more than 4k bytes of data when communicating with an Oracle database. The Oracle large object adapter works around this limitation using an Oracle-specific API.
|PORT||The TCP/IP port to which this engine will listen. This should be set to a port number that is available on the server.||7520|
The URL to an application server. Used when creating an initial context.
|RANDOM_JOB_SELECT||(Available in Flux 8.0.13)|
A Flux cluster consists of multiple Flux engines. If workflows are configured to run on any available Flux engine, and the schedules for many jobs are identical (like run 500 jobs all at 12 noon) there can be database contention over claiming jobs for execution. This is due to each Flux engine fetcheing the same jobs from the Flux database in the exact same order as the other engines. This specific performance improvement returns the selected workflows in random order (within the time they are expected to fire) to each engine to reduce the contention over job selection. This yields a noticeable improvement in overall performance for Flux clusters.
Note: this improvement is not used if Flux Cluster_Networking is enabled. In such instances the cluster networking itself reduces such job contention.
Indicates whether workflow run history information is recorded to the database; disabling run history reduces database activity. Run histories include execution times for workflows and actions as well as workflow metrics. These metrics include counts of successful and unsuccessful workflow runs.
Used to populate the workflow namespace with configuration properties that apply to one engine only. These configuration properties can change on the fly, that is, dynamically. A union of the cluster-wide runtime configuration properties and the per-engine runtime configuration properties is formed to govern engine behavior. If the union creates a conflict, the per-engine properties take precedence.
A default runtime configuration that allows at most one workflow to run at the same time and defines a default error handler that retries failed actions up to five times, with one minute delays after a failure. This default error handler gives up after the fifth failure and places the workflow in the ERROR super-state and the PAUSED sub-state.
Starting with Flux 8.0.11
NOTE: Starting with Flux 8.0.11 the default error handler is installed that displays the action that failed and the error message generated by the failed action. The workflow is set to the FAILED/ERROR state in the operations console. No retry is attempted.
This default runtime configuration is immutable. You cannot make any changes to it.
The path to a runtime configuration file. Used to load a runtime configuration from a file instead of using the appropriate APIs.
A time expression that specifies the frequency with which the runtime configuration file is reloaded automatically. Using this configuration property, if changes are made to the runtime configuration file, they can be imported into the engine automatically.
+1m (one minute)
Specifies the scripting languages that are registered with the Flux engine. Scripting languages are used for pre- and post-processing in triggers and actions.
null (BeanShell is assumed to be the default language in this case)
Indicates whether security is enabled.
|SERVER||Indicates whether this engine will be created as a server that can be contacted over the network.||true|
|SSL||Indicates whether SSL is enabled on the server. This allows secure communication between the engine and all clients, including the Operations Console.||true|
The maximum amount of time the engine sleeps when it has nothing to do.
+3m (three minutes)
The minimum amount of time the engine sleeps before polling the database to determine if more work is available. This option is a temporary workaround to reduce database activity until a better solution can be implemented.
+1s (one second)
The name of the logger used when logging system information.
Command to obtain system resource information such as CPU and memory usage. Standard output generated by the command is used to update system resource values. The system resource command is executed at the intervals defined by the system resource interval configuration property.
none (the system resource command is not used by default)
A time expression specifying how often the SYSTEM_RESOURCE_COMMAND should be executed.
"" (not set by default)
The prefix for all of the Flux database tables.
A JDBC database URL. If DATABASE_TYPE is DERBY or H2, this property is ignored.
The name used to locate a Work Manager resource, to which the engine delegates background processing instead of using threads created directly by the engine. If this property is not defined or does not refer to a valid Work Manager resource, the engine delegates background processing to threads that it creates directly.
Disabling the SERVER option by setting:
In your engine configuration allows you to prevent Flux from accepting any remote connections to the engine. This can be useful when the engine needs to be "locked down" and unavailable to end users, or if there are restrictions on the network that require Flux to avoid opening ports for communication.
Note, however, that this property can only be set if your engine is created programmatically (using the Flux Java APIs). If your engine is started from the command line, from a service on the Operating System, or from a batch or shell script, the SERVER property will always be enabled even if you have explicitly disabled it in the configuration.
The reason for this is that the engine, if created through code, can still be interacted with locally (using the Java API). If the engine is created from a command line, service, or script, however, it will not be able to accept any work or input from a user (since the only way to send work to an engine is to access the engine over the network). Because of this, if you have a need to disable the SERVER option, you will need to be sure that the engine is created using the Java APIs.
Note also that to completely disable network communications on the engine, you will need to disable cluster networking as well, by setting:
In your engine configuration.
SERVER and SECURITY_ENABLED
If you have enabled security on your engine by setting:
The engine will automatically be accessible over the network. It is not necessary to enable the SERVER property; if you do attempt to set SERVER=true while security is enabled, the engine will encounter an error on startup and will fail to start until you remove the SERVER property.
If you set SERVER=false, but security is enabled, then the SERVER property is ignored and the engine will automatically enable remote communication. It is not possible to disable remote communication with security enabled.
This also means that a secured server will always accept remote communication (provided that valid credentials are specified) and remote communication does not need to be explicitly enabled.
For more information on remote communication and security, see Security.
Configuring Flux to Use Your Database
See Configuring Flux to run with Databases for instructions on setting up the Flux engine to point to your database.
Configuring the System Resource Command
If you have the SYSTEM_RESOURCE_COMMAND property set, Flux can use the output of the command invoked to gather information about the current resources on the system (including memory and CPU statistics) and make scheduling decisions based on that information.
When you invoke the system resource command, it must display the following values to standard output, on a single line, separated by tab characters, in this order:
- Percentage of CPU usage
- Total physical memory
- Free physical memory
- Total virtual memory
- Free virtual memory
The output of the monitor command, therefore, might look something like:
To see a sample script you can adapt for your own system resource command, see the examples/software_developers/system_resource_monitoring directory under your Flux installation directory. This directory contains example scripts for both Windows and Unix environments that you can use to quickly configure the command on your own system.
Once you've enabled the system resource monitor, Flux will automatically use the CPU information it retrieves to make scheduling decisions across the cluster. As long as you have two or more engines running in the cluster and all of the engines have the system resource command enabled, Flux can automatically assign workflows to the engine with the lowest CPU usage (and therefore the engine with the most available resources for running a new workflow).