Child pages
  • Performance Related Configuration Settings

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The following settings have a significant impact on Flux performance. Setting them to these recommendations optimizes the engine for performance, at the expense of functionality, loss of history, or logging details.

    • If you are not seeing the amount of concurrency you expect (always <=10 but you are expecting more) - consider that having a runtime configuration file (that is referenced within the engine configuration properties file) impacts where concurrency is set for a Flux engine. 
      • First, make absolutely sure that the engine configuration properties file entry for RUNTIME_CONFIGURATION_FILE points to a valid and accessible runtime configuration properties file.
    • If a runtime configuration file is defined in the engine configuration, concurrency is set using CONCURRENCY_THROTTLE in the runtime configuration file. 
    • If no runtime configuration file is configured, concurrency is set using CONCURRENCY_LEVEL in the engine configuration file. 
    • If a runtime configuration file is being used, and no CONCURRENCY_THROTTLE is set within that runtime configuration file, the engine will run with it's concurrency set to 10 regardless of what the engine's CONCURRENCY_LEVEL is set to.
  • Check the size of the FLUX_AUDIT_TRAIL table. If it is > 100,000 rows consider truncating it and reducing the number of audit trail records being written by using the AUDIT_TRAIL_FILTER setting. Large audit trail tables can slow overall Flux performance.
  • Check that the FLUX_CLUSTER table does not have old or duplicated entries. Truncating the FLUX_CLUSTER table while the engine is down, and then restarting the engine, can improve performance if old entries are present.
  • Set SERVER=false (In most cases this is not an option since this disables the Flux engine's REST API, Flux agents, and the operations console web application)
  • Set INTERNAL_LOGGER_LEVEL=INFO (to reduce the amount of logging)
  • Make sure - if using the default Flux internal logger - that it is set as follows: logger_types.0=INTERNAL_ASYNCHRONOUS. Having it set to INTERNAL_SYNCHRONOUS will make blocking-writes to the log, which is useful in developing and debugging workflows but not in production.
  • Set CLUSTER_NETWORKING_ENABLED=false (but note that this disables agents and disables the bytes sent/received displayed on the console while file transfers are executing).
  • Set CACHE_TYPE=NONE to stop using the cache for workflows.
  • Minimize or eliminate the use of prescripts and postscripts since these involve the interactive execution of code within workflows and reduce performance
  • Set RUN_HISTORY_ENABLED=false to turn off run history data collection
  • Set FLOW_CHART_DEADLINES_ENABLED=false to turn off checking for flowchart deadlines
  • Set AUDIT_TRAIL_FILTER.0= (and no other audit trail filters are present or all others are commented out)
  • Reduce the property SYSTEM_DELAY=+3m to SYSTEM_DELAY=+5s.  The SYSTEM_DELAY is the maximum amount of time the engine sleeps when it has nothing to do (i.e., it has completed all workflows or it has just started up). Reducing this time makes the system more aggressive in looking for work to process in between intervals when all work has finished.
  • (For builds of 8.0.13) If potentially executing many workflows that can be shared across a Flux cluster, and those workflows initiate at the same time based on their timer triggers, set the property RANDOM_JOB_SELECT=TRUE. This property randomizes the returned rows to reduce the contention between engines. If the workflows are returned to both engines in the same order at the same time, the engines attempt to process the same workflows at the same time - which Flux addresses by stopping the second (or greater) engine from claiming the workflow. This creates a lot of needless database contention. The reason RANDOM_JOB_SELECT is not by default set to TRUE is that some customers reply on the order returned for their application-specific processing.

...