Skip to end of metadata
Go to start of metadata

Overview

The engine configuration defines both how your engine runs, and how it stores and maintains data from the Flux engine (including workflow and user data).

The engine configuration settings can be configured using the Operations Console, in a .properties file, or, if you are starting the engine using Java code, a flux.Configuration object. These configuration settings cannot be changed after the engine has started. For a comprehensive list of static configuration options, see the Javadoc for the flux.Configuration option (Javadocs are located in the /doc/javadoc directory under your Flux installation directory).

Engine behavior is also controlled by the Runtime Configuration and Runtime Variables. The runtime configuration and the engine configuration are used together to define how the engine operates.

List of Configuration Properties

The following table contains a list of configuration properties that can be set in your engine configuration, along with a description of each, and the default value (if any) that is used if you do not set the configuration property yourself.

Property

Description

Default Value

AGENT_FAILOVER_TIME_WINDOW

A time expression that specifies the duration of time between agent connectivity tests. If an agent is busy and cannot be reached during this time period, the process assigned to the unreachable agent will be reassigned to the next available agent in the pool.

+15m

AUDIT_TRAIL_EXPIRATION

A time expression indicating how old individual audit trail, run average, and run history entries must be before they are automatically pruned. Expired entries are automatically deleted from the database. A null audit trail expiration indicates that these entries are never deleted from the database.

If your workflows use the Audit Trail Trigger, take caution when setting this property to ensure that your triggers will have sufficient time to record events before they are deleted from the database.

+1H (when using the H2 in-memory database), OR

+w (for all other database types)

AUDIT_TRAIL_FILTER

A list of audit trail events to record to the audit trail. If this audit trail filter is not set or empty, all audit trail events are recorded to the audit trail.

For usage instructions and examples, see Logging and Audit Trail.

none (by default, all audit trail events are recorded)

AUDIT_TRAIL_LOGGER

The name of the logger used when logging audit trail information.

audit_trail

AUDIT_TRAIL_ROLLBACK_RECORDINGS_ENABLED

Indicates whether actions that execute in a workflow that are later rolled back in the curent database transaction are still recorded to the audit trail. Useful for knowing what actions led up to an error, before the error caused the current database transaction to be rolled back.

false

CACHE_SIZE

The initial size of the workflow cache. The cache size determines the number of workflows that can be stored in the cache.

Flux will dynamically increase the size of the cache as needed to accommodate larger loads of workflows that arrive.

200

CACHE_TYPE

This configuration property controls how workflows are cached.

There are two possible settings for this property:

  • LOCAL: Workflows are only cached locally.
  • NONE: Workflows are not cached at all and are pulled directly from the database

Depending upon the nature of your workflows, there are instances where LOCAL cache performs more slowly than NONE. This is especially true with very large workflows involving many actions and variables.

NONE

CLIENT_LOGGER

The name of the logger used when logging client-side information.

client

CLUSTER_NETWORKING_ENABLED

Indicates whether networking is enabled in the engine instance when it participates in a cluster. Networking is not required for clustering, but networking increases responsiveness among engine instances across the cluster. By disabling cluster networking, the engine instance still participates in a cluster, but all its communication with other engine instances is performed through the database server.

The file transfer monitor user interface and the execution of Flux agents depend on cluster networking being enabled. These features will be disabled when cluster networking is turned off.

Note that it does take longer to remove workflows from the engine if cluster_networking is enabled. A "You are attempting to modify workflow ..." error may occur if workflows are being added to the engine, immediately executed, and then resubmitted using the same names. In such instances it is recommended that cluster_networking be disabled.

true

CONCURRENCY_LEVEL

The concurrency throttle limit in the root node of the runtime configuration tree. This limit controls the number of flow contexts that may execute at one time across the entire engine.

In essence, this configuration property provides a shortcut to the concurrency throttle limit in the root node of the runtime configuration tree, for the sake of convenience.

This property is only used if a runtime configuration file is not being used. If a runtime configuration file is present, the Flux engine will use the CONCURRENCY_THROTTLE specified in the runtime configuration, or a default of 10 if none is specified.

10

DAILY_MAINTENANCE_TIME

A time expression specifying the time at which the engine performs once-a-day maintenance on the Flux database tables. For example, to run daily maintenance each morning at 04:30, set this configuration property to the Cron-style time expression "0 0 30 4". The time zone is always the local time zone.

Caution

When choosing the time expression used, make sure maintenance is only performed exactly once per day. Do not run maintenance multiple times a day as this may interfere with Flux processing.

0 0 0 2 (By default, maintenance is done at 02:00 each morning)

DATA_SOURCE

The JNDI name used to lookup an application server's database connection pool.

none (you must explicitly set this property in order to use a data source)

DATA_SOURCE_AUTO_COMMIT_CHECK

Indicates whether to disable auto-commit on database connections obtained through a data source. If this property is enabled, the engine will check every connection retrieved from the data source for an auto-commit value of "false". If the connection has an auto-commit value of "true", the engine will log a warning and automatically attempt to disable auto-commit on the connection.

If this property is disabled, the engine will not attempt to check data source connections for the auto-commit value.

true

DATA_SOURCE_CACHING

Indicates whether data source objects are saved and re-used later. Such caching of data source objects reduces warning messages emitted by WebSphere 5.0. Unfortunately, this caching may also reduce the ability of an application server to perform clustering. If possible, it is preferred to keep this configuration property at its default setting of false, which means that data source objects are not cached.

false

DATA_SOURCE_PASSWORD

The password used to retrieve a database connection from an application server data source. This property is only used if it is not null and not the empty string, the data source username is not null and is not the empty string, and a data source has been set.

"" (empty string)

DATA_SOURCE_USER_TRANSACTION_CLIENT_JNDI_NAME

The JNDI name of a javax.transaction.UserTransaction object that should be used for client calls into the Engine. User transactions are used to guarantee transaction integrity when the engine's transactions need to be integrated with transactions from other software systems.

This property is only used when DATA_SOURCE_USER_TRANSACTIONS is enabled.

java:comp/UserTransaction

DATA_SOURCE_USER_TRANSACTION_SERVER_JNDI_NAME

The JNDI name of a javax.transaction.UserTransaction object that should be used for workflows and background tasks running on the engine. User transactions are used to guarantee transaction integrity when the engine's transactions need to be integrated with transactions from other software systems.

This property is only used when DATA_SOURCE_USER_TRANSACTIONS is enabled.

java:comp/UserTransaction

DATA_SOURCE_USER_TRANSACTION_TIMEOUT

A time expression specifying the timeout for all user transactions that the engine uses. If this property is null, Flux will not set timeouts for user transactions.

This property is only used when DATA_SOURCE_USER_TRANSACTIONS is enabled.

+d (one day)

DATA_SOURCE_USER_TRANSACTIONS

Indicates whether a User Transaction object manages database transactions that are retrieved from a data source. If true, a javax.transaction.UserTransaction object is used to manage transactions. If false, Flux manages transactions.

FALSE

DATA_SOURCE_USERNAME

The username used to retrieve a database connection from an application server data source. This property is only used if it is not null and not the empty string and a data source has been set.

"" (empty string)

DATABASE_PROPERTIES

The full path to a database properties file. Database properties allow you to specify how Flux should connect to the database, to rename database tables and columns, to reassign a standard SQL type, and to provide initialization SQL statements to database connections that are created to directly access the database.

For complete documentation on this property, see Databases.

none

DATABASE_TYPE

The type of database that the engine should use.

NOTE: If the DATABASE_TYPE is set to Derby or H2, you will not be able to use clustering or failover with this engine.

DERBY

DRIVER

The class name of a JDBC driver. If DATABASE_TYPE is DERBY or H2, this property is ignored (as Flux automatically manages drivers for those database types).

none

ENGINE_CONTENTS_SUMMARY_FREQUENCY

A time expression that specifies how often engine contents status audit trail events are generated. The engine contents summary audit trail event provides information on all of the workflows, messages, and message publishers that exist on the engine. Although this property allows you to adjust the frequency of these events, it is not possible to disable the engine contents summary completely.

The minimum frequency is 5 minutes.

+1d (one day)

FAILOVER_TIME_WINDOW

A time expression that specifies how often the engine checks to see whether other engines in the cluster have failed. This is used for failover purposes to determine whether any workflows have prematurely stopped running and must be restarted on another engine.

Note that if an engine goes down and comes back up, the number of reported firing workflows can be up to double what would normally be expected of that recovering engine. This is because the recovering engine, upon startup, will schedule and begin firing a number of workflows equal to its concurrency level. But the engine will have already marked workflows as firing before it went down. These workflows will not be recovered, and their firing state changed, until the FAILOVER_TIME_WINDOW has been exceeded. Once the FAILOVER_TIME_WINDOW has passed the amount of concurrently firing workflows will return to the expected concurrency level. 

+3m (three minutes)

Make Sure

Ensure that all servers in the Flux cluster have the same time value set. Use a network time service to synch times between Flux servers. To not do so can cause Flux actions to fire multiple times if the time drifts between servers.

 

 

FAIRNESS_TIME_WINDOW

A time expression that specifies how often Flux should temporarily increase workflow priorities to prevent lower-priority workflows from being starved (not allowed to run) by higher-priority workflows. A non-running workflow's priority will be temporary updated each time Flux performs the fairness check. The workflow's priority may be temporarily increased several times before it is able to run. Once the workflow does run, its priority will be reset to its original value and it will begin waiting again for the next opportunity to execute.

A workflow's permanent priority is called its "priority". If the priority is temporarily elevated by the fairness, that elevated priority is called its "effective priority".

For additional information, see the runtime configuration priority setting.

none (fairness checks are not performed by default – setting a value for this property will automatically enable them)

FILE_TRANSFER_DEBUG

Indicates whether debugging is enabled for file transfers. If this property is enabled, all interaction between Flux and any remote file hosts will be logged to the Flux system log. When enabled, this property requires the INTERNAL_LOGGER_LEVEL to be set at FINE logging level or greater.

false

FLOW_CHART_DEADLINES_ENABLED

Indicates whether workflow deadlines (SLAs or Service Level Agreements) are enabled; disabling deadlines reduces database activity. When this property is enabled, you can set deadlines on individual workflows. If a workflow exceeds its deadline, it will publish an audit trail event indicating that the deadline has passed. The Operations Console will also graphically indicate any workflows that are approaching (or have exceeded) their deadlines.

true

FLOW_CHART_LOGGER

The name of the logger used when logging information during workflow execution.

flow_chart

HEARTBEAT_FREQUENCY

A time expression that specifies the frequency with which heartbeat audit trail events are generated. A heartbeat audit trail event provides information on the health of the engine. Although this property allows you to adjust the frequency of these events, it is not possible to disable heartbeats completely.

The minimum frequency is 5 seconds.

+1H (one hour)

HOST

The host name or IP address that the engine should bind to. If there are multiple host names / IP addresses available on the server where the engine is running, this property allows you to specify which one Flux should use for network requests.

If you are using this configuration to look up an existing engine, this property allows you to specify the host name where the engine is running.

Flux will automatically select a host name from the available names on the system

ID_DESCRIPTION

A description of the engine instance intended for human consumption.

none

ID_NAME

The case-insensitive name of the engine instance. Used to differentiate Flux engine instances in the logs and audit trail. If the name is not set explicitly in a configuration, the configuration will generate a name using:

  1. The host name of the local computer but not "localhost". For example, "foo" or "foo.bar.com".
  2. Failing that, the IP address of the local computer but not "127.0.0.1". For example, "12.34.56.78".
  3. Failing that, "unknown".
    If there is more than one active Flux engine in a JVM, each generated name is made unique. To make a generated name unique, a ":" symbol is first appended to the name that is generated as described above. A number, starting from one, is then appended to ensure that the generated name is unique within the JVM.

determined on engine startup (see description column)

INITIAL_CONTEXT_FACTORY

The JNDI name of an application server's initial context factory.

none

INITIAL_CONTEXT_PASSWORD

The password used to create an application server initial context when looking up objects in an application server's JNDI tree (like data source, EJB, or JMS information). This property is only used if it is not null and not the empty string and the INITIAL_CONTEXT_USERNAME configuration property is not null and is not the empty string.

"" (empty string)

INITIAL_CONTEXT_USERNAME

The username used to create an application server initial context when looking up objects in an application server's JNDI tree (like data source, EJB, or JMS information). This property is only used if it is not null and not the empty string.

"" (empty string)

INTERNAL_LOGGER_FILE_DIRECTORY

The directory where log files are placed when using the internal logger.

. (equivalent to the system property "user.dir", typically the directory where the engine or JVM was started)

INTERNAL_LOGGER_FILE_ROTATION_SIZE

The maximum size to which a log file for the internal asynchronous and internal synchronous loggers can grow before it is rotated. At most, there are seven log files, which include the current log file and up to six previously rotated log files. The rotation size is specified in bytes.

10485760 (ten megabytes, 10 * 1024 * 1024)

INTERNAL_LOGGER_LEVEL

The logging level used for the internal asynchronous and internal synchronous loggers. This property does not affect the logging level for any other loggers.

INFO

JDBC_CONNECTION_RECYCLE_FREQUENCY

When Flux is configured to obtain database connections directly through JDBC (rather than a data source), it creates a connection pool and periodically recycles connections in the pool. This property is a time expression that indicates how often Flux should recycle database connections.

The minimum recycle frequency is 15 minutes, and the maximum is 90 days.

+1H (one hour)

JDBC_ENCRYPTED_PASSWORD

The encrypted password used to obtain database connections when Flux is configured to obtain database connections directly through JDBC (rather than a data source). This property is only used if it is not null and not the empty string, the JDBC_USERNAME configuration property is not null and is not the empty string, and the JDBC_PASSWORD configuration property is null or the empty string.

Passwords can be encrypted using the encrypt password command line utility.

"" (empty string)

JDBC_PASSWORD

The password used to obtain database connections when Flux is configured to obtain database connections directly through JDBC (rather than a data source). This property is only used if it is not null and not the empty string and the JDBC_USERNAME configuration property is not null and is not the empty string.

Must be supplied in cleartext format.

none

JDBC_USERNAME

The username used to obtain database connections when Flux is configured to obtain database connections directly through JDBC (rather than a data source). This property is only used if it is not null and not the empty string.

sa

KEEP_COMPLETED_ON_ENGINE

(Available in Flux 8.0.13)

Keeps completed workflows on the engine and in the Flux Ready Table. Useful for development and testing of workflows. Note that if running thousands of workflows to completion per day, this option can cause performance slowdowns if the completed workflows are not periodically removed.

false

LOG_EXPIRATION

A time expression indicating how old log entries in the database must be before they are pruned (deleted) from the database. At the DAILY_MAINTENANCE_TIME each day, Flux will find and prune all entries older than this time expression.

If this property is null, Flux will not attempt to prune log entries in the database.

+w (one week). If the DATABASE_TYPE is H2, however, the default value is ?+1H (one hour).

LOGGER_TYPE

Indicates which logging system Flux should use. This property should only be set when a single logging system is used on the engine; for multiple loggers, use the LOGGER_TYPES property instead.

INTERNAL_ASYNCHRONOUS

LOGGER_TYPES

Indicates which logging systems Flux should use. This property should only be set when multiple logging systems are to be used on the engine (for example, this could be configured to use both log4j and the internal asynchronous logger to log messages).

For usage instructions, see Logging and Audit Trail.

INTERNAL_ASYNCHRONOUS

MAX_CONNECTIONS

The maximum number of database connections that the engine should be allowed to have open at once.

For more information on the MAX_CONNECTIONS, including recommended settings, see Max Connections and Concurrency Level / Concurrency Throttle.

15

ORACLE_LARGE_OBJECT_ADAPTER

By default, Flux uses standard JDBC APIs for persisting large binary data and characters to the database. Due to an Oracle limitation, however, standard JDBC cannot persist more than 4k bytes of data when communicating with an Oracle database. The Oracle large object adapter works around this limitation using an Oracle-specific API.

This configuration property accepts the name of an adapter class that uses database- and driver-specific APIs to overcome this Oracle limitation. The adapter class specified must implement the flux.OracleLargeObjectAdapter interface.

For your convenience, Flux provides several such adapter classes by default. They can be used either when the Oracle JDBC driver is used directly, or when an application server driver wraps the Oracle driver (an Oracle driver supplied by Oracle Corporation must be used, however, either directly or indirectly). The class names for these drivers are listed as constants in the Javadoc documentation for the flux.OracleLargeObjectAdapter interface.

If you prefer, you can provide your own adapter. It must implement the flux.OracleLargeObjectAdapter interface. Your custom adapter can be used to support non-Oracle drivers or other application servers that provide JDBC drivers that wrap the Oracle JDBC drivers.An Oracle Large Object Adapter must be defined when using an Oracle data source.

none

PORTThe TCP/IP port to which this engine will listen. This should be set to a port number that is available on the server.7520

PROVIDER_URL

The URL to an application server. Used when creating an initial context.

none

RANDOM_JOB_SELECT(Available in Flux 8.0.13)

A Flux cluster consists of multiple Flux engines. If workflows are configured to run on any available Flux engine, and the schedules for many jobs are identical (like run 500 jobs all at 12 noon) there can be database contention over claiming jobs for execution. This is due to each Flux engine fetcheing the same jobs from the Flux database in the exact same order as the other engines. This specific performance improvement returns the selected workflows in random order (within the time they are expected to fire) to each engine to reduce the contention over job selection. This yields a noticeable improvement in overall performance for Flux clusters.

Note: this improvement is not used if Flux Cluster_Networking is enabled. In such instances the cluster networking itself reduces such job contention.

false

RUN_HISTORY_ENABLED

Indicates whether workflow run history information is recorded to the database; disabling run history reduces database activity. Run histories include execution times for workflows and actions as well as workflow metrics. These metrics include counts of successful and unsuccessful workflow runs.

true

RUNTIME_CONFIGURATION

Used to populate the workflow namespace with configuration properties that apply to one engine only. These configuration properties can change on the fly, that is, dynamically. A union of the cluster-wide runtime configuration properties and the per-engine runtime configuration properties is formed to govern engine behavior. If the union creates a conflict, the per-engine properties take precedence.

A default runtime configuration that allows at most one workflow to run at the same time and defines a default error handler that retries failed actions up to five times, with one minute delays after a failure. This default error handler gives up after the fifth failure and places the workflow in the ERROR super-state and the PAUSED sub-state.

Starting with Flux 8.0.11

NOTE: Starting with Flux 8.0.11 the default error handler is installed that displays the action that failed and the error message generated by the failed action. The workflow is set to the FAILED/ERROR state in the operations console. No retry is attempted.

This default runtime configuration is immutable. You cannot make any changes to it.

RUNTIME_CONFIGURATION_FILE

The path to a runtime configuration file. Used to load a runtime configuration from a file instead of using the appropriate APIs.

See Runtime Configuration and Runtime Variables for usage instructions.

none

RUNTIME_CONFIGURATION_FILE_REFRESH_FREQUENCY

A time expression that specifies the frequency with which the runtime configuration file is reloaded automatically. Using this configuration property, if changes are made to the runtime configuration file, they can be imported into the engine automatically.

+1m (one minute)

SCRIPTING_LANGUAGES

Specifies the scripting languages that are registered with the Flux engine. Scripting languages are used for pre- and post-processing in triggers and actions.
If this configuration property is null, BeanShell is used as the default scripting language.

See Prescripts, Postscripts And Scripting in General for more information and usage.

null (BeanShell is assumed to be the default language in this case)

SECURITY_ENABLED

Indicates whether security is enabled.

An engine that is created with security enabled can accept remote connections from clients and users. Either this property or SERVER must be set to true (see below) in order to connect to the engine remotely (using the Operations, the Java APIs, or any other client).

true

SERVERIndicates whether this engine will be created as a server that can be contacted over the network.true
SSLIndicates whether SSL is enabled on the server. This allows secure communication between the engine and all clients, including the Operations Console.true

SYSTEM_DELAY

The maximum amount of time the engine sleeps when it has nothing to do.

+3m (three minutes)

SYSTEM_DELAY_MINIMUM

The minimum amount of time the engine sleeps before polling the database to determine if more work is available. This option is a temporary workaround to reduce database activity until a better solution can be implemented.

+1s (one second)

SYSTEM_LOGGER

The name of the logger used when logging system information.

system

SYSTEM_RESOURCE_COMMAND

Command to obtain system resource information such as CPU and memory usage. Standard output generated by the command is used to update system resource values. The system resource command is executed at the intervals defined by the system resource interval configuration property.

This should either be the full path to a command, script, or executable on the server, or the name of a command, script, or executable that is available from the system PATH.

See below for more information on the type and syntax of output that is expected to be returned from this command.

none (the system resource command is not used by default)

SYSTEM_RESOURCE_INTERVAL

A time expression specifying how often the SYSTEM_RESOURCE_COMMAND should be executed.

"" (not set by default)

TABLE_PREFIX

The prefix for all of the Flux database tables.

FLUX_

URL

A JDBC database URL. If DATABASE_TYPE is DERBY or H2, this property is ignored.

none

WORK_MANAGER_RESOURCE_NAME

The name used to locate a Work Manager resource, to which the engine delegates background processing instead of using threads created directly by the engine. If this property is not defined or does not refer to a valid Work Manager resource, the engine delegates background processing to threads that it creates directly.

For WebSphere and WebLogic, this name refers to a CommonJ WorkManager JNDI name configured in WebSphere or WebLogic. For JBoss, this name refers to a work manager MBean (for example: "jboss.jca:service=WorkManager") configured in JBoss. For GlassFish, this name refers to a thread pool (for example: "thread-pool-1") configured in GlassFish.

none

Disabling SERVER

Disabling the SERVER option by setting:

SERVER=false

In your engine configuration allows you to prevent Flux from accepting any remote connections to the engine. This can be useful when the engine needs to be "locked down" and unavailable to end users, or if there are restrictions on the network that require Flux to avoid opening ports for communication.

Note, however, that this property can only be set if your engine is created programmatically (using the Flux Java APIs). If your engine is started from the command line, from a service on the Operating System, or from a batch or shell script, the SERVER property will always be enabled even if you have explicitly disabled it in the configuration.

The reason for this is that the engine, if created through code, can still be interacted with locally (using the Java API). If the engine is created from a command line, service, or script, however, it will not be able to accept any work or input from a user (since the only way to send work to an engine is to access the engine over the network). Because of this, if you have a need to disable the SERVER option, you will need to be sure that the engine is created using the Java APIs.

Note also that to completely disable network communications on the engine, you will need to disable cluster networking as well, by setting:

CLUSTER_NETWORKING_ENABLED=false

In your engine configuration.

SERVER and SECURITY_ENABLED

If you have enabled security on your engine by setting:

SECURITY_ENABLED=true

The engine will automatically be accessible over the network. It is not necessary to enable the SERVER property; if you do attempt to set SERVER=true while security is enabled, the engine will encounter an error on startup and will fail to start until you remove the SERVER property.

If you set SERVER=false, but security is enabled, then the SERVER property is ignored and the engine will automatically enable remote communication. It is not possible to disable remote communication with security enabled.

This also means that a secured server will always accept remote communication (provided that valid credentials are specified) and remote communication does not need to be explicitly enabled.

For more information on remote communication and security, see Security.

Configuring Flux to Use Your Database

See Configuring Flux to run with Databases for instructions on setting up the Flux engine to point to your database.

Configuring the System Resource Command

If you have the SYSTEM_RESOURCE_COMMAND property set, Flux can use the output of the command invoked to gather information about the current resources on the system (including memory and CPU statistics) and make scheduling decisions based on that information.

When you invoke the system resource command, it must display the following values to standard output, on a single line, separated by tab characters, in this order:

  1. Percentage of CPU usage
  2. Total physical memory
  3. Free physical memory
  4. Total virtual memory
  5. Free virtual memory

The output of the monitor command, therefore, might look something like:

10.000000        1051120 306356  2097024 2065060

To see a sample script you can adapt for your own system resource command, see the examples/software_developers/system_resource_monitoring directory under your Flux installation directory. This directory contains example scripts for both Windows and Unix environments that you can use to quickly configure the command on your own system.

Once you've enabled the system resource monitor, Flux will automatically use the CPU information it retrieves to make scheduling decisions across the cluster. As long as you have two or more engines running in the cluster and all of the engines have the system resource command enabled, Flux can automatically assign workflows to the engine with the lowest CPU usage (and therefore the engine with the most available resources for running a new workflow).

  • No labels