Skip to end of metadata
Go to start of metadata

The following example workflows are provided to simplify getting started with Flux. These workflows are easily modified and extended to meet your requirements.

Action prescripts and postscripts are used in these flows to provide status messaging and details of the file name being moved. See example prescript here of updating the status message in the Operations Console during a file copy: Prescript Example.png

Output File Information from a Directory (Basic)

This workflow scans a directory and outputs information about all the files it finds. This example illustrates how to query a directory, iterate through the files, and format each file's information into a console message.

 

Download here:  Output File Information.ffc

Bulk File Copy (Basic)

This workflow scans a directory and copies all the found files. The bulk file copy is useful where all the files have to be copied at a point in time. If this bulk file copy fails, rerunning the flow will copy all the files over again.

This example assumes that a runtime configuration file has been defined, and that runtime configuration properties to identify the file source directory and target directory specified as follows:

 

Required Runtime Configuration File Properties

Download here: Bulk-Copy.ffc

File Copy using Java Code (Basic)

This example illustrates using Java code to do the following:

  • create a Flux flowchart, 
  • create the criteria for files we want to include, exclude, and filter out using a regular expression filter in the file copy,
  • set the target directory,
  • define a renamer to rename files once they are copied,
  • create a file copy action within the flowchart and populate it with the above criteria,
  • submit the flowchart to the engine for execution, and then 
  • stop (i.e., dispose) the engine, Here is the Javadoc for the Flux engine.

The complete code for this example is in the examples\software_developers\file_copy directory of a Flux installation.

 

Java Code Example  Expand source

Mass File Copy and Prepend Today's Date to Each Copied File (Basic)

This workflow scans a directory and copies all the found files. The workflow consists of just a single File Copy Action. A Regular Expression Renamer is configured to prepend the date to each copied file.

Note in the illustration below that the From Pattern is (.*). This regular expression captures the file name into a group, later referenced in the To Pattern as \1 (Since it is the first and only regex group captured).

The To Pattern is ${$date yyyy-MM-dd}-\1. This pattern prepends the date in a form such as 2014-Jun-18, with a dash '-', in front of each file name copied from the current directory into c:\test.

                               

Download here: MassFileCopyWithRegexToPrependDate.ffc

Generic File Move, Copy, Rename, Delete (Basic)

This workflow creates a file, tests for its existence, moves it somewhere, copies it back to where it was moved from, renames it, and deletes both files and tests to make sure they both no longer exist.

This example assumes that a runtime configuration file has been defined, and that runtime configuration properties to identify the file source directory and target directory specified as follows:

 

Required Runtime Configuration File Properties

Download here: FLUX-SAMPLES-FileActions.ffc

Read a CSV File and Print Result (Basic)

This example uses the CsvJdbc library to connect to a CSV file located in the local system, and perform a query. 

This example assumes that you have the 'csvjdbc-1.0-20.jar' in your Flux Classpath; and a CSV file named 'customers.csv' in the path '/Users/Documents'. Using Flux's built-in Database Query action it creates a connection to the location of the file and then performs the query set in the 'Query' property of the Database Query action. 

The JDBC URL property specifies the path to the file, to change the path to point to another location, just append the path to 'jdbc:relique:csv:' as shown below:

In the Query we specify the file name (without file extension):

Where 'customers' is the name of our CSV file. The output should look like:

Download here: FLUX-SAMPLES-CSVJDBC.ffccustomers.csvcsvjdbc-1.0-20.jar

Simple Database Query Using the Database Query Action (Basic)

This example illustrates how to query a database table and output values from the returned results. This example queries the FLUX_FLOW_CHART created within every Flux installation. When the DatabaseQuery Action runs using the query select * from FLUX_FLOW_CHART, it returns one database row at a time and passes that to the Console Action. The Console Action outputs the ActionResult and also the contents of the first column (in this case a numeric flowchart identifier.



Shown below are the properties for the flow from the Database Query Action to the Console Action (the flow in blue above). Note that the Condition is "Custom". The Database Query Action returns a RESULT object that contains the fields "row" and "result". The "result" field, if set to true, indicates there are additional rows in the query. The "row" field contains the database row contents. So while the RESULT.result field is true, the Console Action is sent a row to print out.  


Here the RESULT object is printed out, and the first column of the RESULT's row. Printing out the RESULT object just prints out the object's name and identifier, which is generally only useful if one is trying to confirm the type of returned result. Returning the column values in the row is generally much more interesting!

Download here: FLUX-SAMPLES-Simple-DB-Select-using-Database-Query-Action.ffc

Ping a Host and Notify if Down (Basic)

This workflow sends a ping request to a host and email a notification if the host is down.

This example assumes that a shell script has been defined in a directory located at /home/flux/FluxTest/Ping. Modify the Process Action to point to a directory of your choosing or enter the full path to the script or executable in the Command text field. Paths can be relative to the directory was Flux was installed or an absolute path can be used (as in this example).

You will also have to modify the Mail Action to input the correct mail server settings like so:

 

Required Shell Script for Linux/Unix
Required Batch File for Windows

 

Note: You'll need to check that your antivirus software doesn't have any rules blocking SMTP. For example, McAfee Enterprise Virus protection - standard for many corporations - has a default setting that blocks SMTP ports 25 and 587. The default rule needs to be modified to add java.exe and telnet.exe as programs allowed to use the mail ports. This is illustrated in the attached screenshot.


 

Download here: FLUX-SAMPLES-Ping.ffc

Mail Trigger that prints to STDOUT the Subject and Body of the email (Basic)

This workflow polls the 'Inbox' of a test mail user and fires when new mail arrives; it then prints out to STDOUT the email subject and the email body. Export the workflow to the engine and then send an email to 'fluxtemplate@gmail.com' (without quotes) and check the Console Action's output match the subject and body. 

Download here: FLUX-SAMPLES-MailTrigger.ffc

Flow Chart Trigger that waits for another workflow to finish copying files and then iterates through them and prints to STDOUT (Basic)

This workflow waits on a Flow Chart trigger until the workflow defined in the 'Namespace' property finishes and then polls a folder and iterates through the list of files found. The 'Namespace' can also be used to define entire namespaces, specific workflows, etc. as explained in Flow Chart Trigger.

You will need to edit the File Exist trigger to point to a valid directory in your system. 

This example waits for the /Incoming/Daily workflow attached to finish. You will need to change the 'Source' and 'Target' of the File Copy action to valid directories in your system. 

Download here: Daily-FlowChartTrigger.ffcIncoming-Daily.ffc

Using a For Each Collection Element action to iterate through a collection (Basic)

This workflow scans a folder for files, iterates through them and prints out the filenames one at a time.

It uses a File Exist trigger to poll for files, then Runtime Data Mapping to pass the resulting collection to a For Each Collection Element action like so:

We're calling ${i} from the 'Print Filenames' Console action, 'i' is the Loop Index of the For Each Collection Element action, and we can use Variable Substitution to call that variable from anywhere in the workflow (keep in mind that until the For Each Collection Element action runs, that variable is empty). 

You will need to modify the File Exist trigger's source folder to a valid path in your environment. 

Download here: FLUX-SAMPLES-IterateFiles.ffc

Calling a workflow using a FlowChart action (Basic)

This workflow exports a workflow that's in the Workflow Repository under a new name, waits for it to finish, and then continues executing. 

To run this example you'll need to download both workflows and upload them to your Repository. Then you can start the /FLUX-SAMPLES/FlowChartAction workflow which will in turn start the child workflow. 

Download here: FLUX-SAMPLES-FlowChartAction.ffcFLUX-SAMPLES-ChildWorkflow.ffc

File Trigger that Times Out and Sends Notification Email (Basic)

This workflow scans two different folders for files and then copies them to the same location. The File Exist triggers have a timeout – the triggers will poll the source folders and if nothing is found after 15 minutes, the triggers time out and exit. When the triggers time out, they flow into an Email action, which sends a notification email to let you know that the files were not found in the source folder as expected. 

To run this workflow, the sources for the File Exist triggers, the targets for the File Copy actions and the mail user, password and SMTP server settings of the Mail actions need to be updated to suit the environment where the workflow will be run. Sample values are provided for all, to make it easier to understand and change them.

Download here: FLUX-SAMPLES-FileTriggerWithTimeout.ffc

Send Email Reminders that Stop After Receiving a Mail (Basic)

Consider the use case of receiving reminders during the day at specific times. Until you respond to a reminder, it keeps reminding you every 10 minutes (or until it quits after a number of times). This example sends email reminders to users, then waits for 10 minutes and re-sends the reminder. To stop a reminder for a given time of day, the user must send/reply to one of the reminders. Then the next reminder is sent. 

To run this workflow you'll need:

  • Access to a mail server to send the notifications from
  • Access to a sub folder in the sender's mail inbox (where the user's can send a 'stop' email will be stored)

Before you run this workflow you must:

1- Create a subfolder/mailbox in a mail account that you want the Flux workflow to access. In this example the Flux engine is connecting to a Gmail account. The account is configured with a subfolder/label named 'Inboxx'.

2- Once the subfolder is created make sure you create an email rule so that the response emails to from the user being sent reminders are automatically redirected to that folder (e.g., Inboxx). NOTE: this step is the most important, if the mail trigger is watching the Inbox instead of a custom folder it will consider ANY incoming mail as the "stop reminders for this hour" email.

Here are the instructions to create rules for Gmail:
http://www.makeuseof.com/answers/set-rules-move-messages-gmail/

3- Modify (if needed) the times each timer trigger fires, and all the email settings on both the Email triggers (watch for incoming mail from the user being reminded) as well as the Email actions (that send the reminder emails). 

4- Export the workflow to the engine.

Download here: FLUX-SAMPLES-EmailReminder.ffc

Mail Box Connections

Icon

Some mail servers (e.g., Gmail) will complain if you try to connect to the mail server too frequently. This may cause the workflow to fail with the following message:

javax.mail.AuthenticationFailedException: 454 4.7.0 Too many login attempts, please try again later. 67sm2929465qkx.38 - gsmtp

Simply increase the mail trigger polling intervals from the +1m as defined in this example.

 

File Copy that Fails and Exits on Error (Intermediate)

This workflow scans a directory and copies a file at a time. If a file copy fails, the workflow is set to ‘FAILED’ in the Operations Console. From the console, you can either ‘restart’ the workflow to run from the beginning, or ‘recover’ the workflow which starts the flow at the point where the file copy failed.

This example assumes that a runtime configuration file has been defined, and that runtime configuration properties to identify the file source directory and target directory specified as follows:

 

Required Runtime Configuration File Properties

Download here: FLUX-SAMPLES-FileCopy-ExitOnError.ffc

Invoking a Rest Service and Print Result to STDOUT  (Intermediate)

This example illustrates how to invoke a REST service on a remote system using Flux. The Rest Action has only one required property: baseURL, which specifies the location of the HTTP service to invoke. Arguments can be passed to the REST service and the REST service can also return values which can be used in the flow chart.

The above screenshot illustrates a Rest Action which invokes the Yahoo StockQuote service at http://finance.yahoo.com/d/quotes.csv. The service takes two arguments: 1. quotes string argument, “GOOG+YHOO+HPQ+IBM+MSFT+ORCL”, and  2. format string argument "snl1d1t1ohgdr", which are passed to the REST service.

The result returned from the REST service is of type RestAction.RestActionResult. The Rest Action postscript maps result.result to a flow context variable "quote".

Variable substitution is used in the Console Action to display the value of the quote flow context variable; and will display the output on the STDOUT.

Download here: FLUX-SAMPLES-GetStockQuote.ffc

File Copy that Continues on Error (Intermediate)

This workflow scans a directory and copies a file at a time and logs a notification in the event of a file copy failure. The workflow than continues to process the next files.

 

This example assumes that a runtime configuration file has been defined, and that runtime configuration properties to identify the file source directory and target directory specified as follows:

 

Required Runtime Configuration File Properties

Download here: FLUX-SAMPLES-FileCopy-ContinueOnError.ffc

Report or Data Delivery (Intermediate)

This Flux workflow starts with a simple database table that contains the email address, file name, and a timestamp of when the file was sent (initially an empty field).  Every five minutes a timer trigger fires, triggering a database query to search the database for records where the sent timestamp is empty. For each record found, a Flux mail action builds an email using a customizable email template, and attaches the file specified in the record. The mail is sent, and the time that the mail was sent is updated in the database record. In the event of a mail failure or a file not found issue, a notification is sent.

This example assumes that a database table named 'Customers' has been defined with an email address, file path (to the file that needs to be delivered), and timestamp field (all defined as character or varchar fields). 

Download here: FLUX-SAMPLES-DB+Email.ffc

Ping an FTP Host and Test Latency Before Copying (Intermediate - Requires Light Scripting)

This Flux workflow pings an FTP host for latency and perform a conditional file copy if the latency is acceptable. The workflow uses a few simple actions:

  • A Process Action performs the ping operation and results its result to stdout.
  • A Regular Expression Action uses variable substitution to retrieve the results, parses the ping time, and uses a postscript (shown below) to convert the ping time for use in a conditional flow.
  • The flow branches to a file copy (if the regular expression found a matching ping time and it was less than 100ms), or to a delay trigger, to wait 5 minutes and retry (if the ping results could not be parsed – that is, if the ping failed – or the ping took longer than 100ms).

The postscript on the Regular Expression action is necessary only to convert the action's result from array format (which cannot be used in a conditional flow) to a long value, which can:

Source for Creating and Saving a Business Interval  Expand source

Download here: PingHostAndCopyFiles.ffc

Put a Job from the Repository to the Engine with a Set of Supplied Variables (Intermediate - Requires CURL or Java Coding)

Starting with Flux 8.0.10, Flux now exposes a REST API that allows one to submit a workflow from the Flux repository to the Flux engine and initialize that workflow with a set of variables (or more precisely - variable values). This is useful in making workflows reusable by exposing variables such as server names or email addresses and allowing these to be added in at workflow submission time.

The following code snippet assumes a workflow exists in the repository named '/Test', and that it has a console action in its workflow that outputs the variable ${Message}. This code submits 500 instances of this workflow to the engine, and each instance will simply output a number. 

 

Putting a Job with Variables to Flux using Java API  Expand source

 

The following example uses a CURL command - to submit the JSON shown - to submit some order details. The JSON keys and values contain the following:

      • repositoryNamespace: The name of the workflow in the repository
      • engineNamespace: The namespace the workflow should have when submitted to the engine for execution
      • overwrite: true | false - Whether to overwrite the engine namespace if the workflow is already on the engine
      • variables: variable names and their string or boolean values. Should match workflow variables defined for the workflow.

 

Putting a Job using CURL
Sample JSON to Supply Variables

 

Adding and Using a Business Interval (Intermediate - Requires Scripting)

The following script code creates a business interval for use in restricting workflows to only run within specified times and dates. 

Source for Creating and Saving a Business Interval  Expand source

Create a workflow and then add a Null Action to the workflow. Take the above code and paste it into the Prescript for the Null Action and then export the workflow to the engine. Flux will run the above script and create the business intervals/calendars defined and add them into the Flux Repository, as illustrated below:

 After this business interval is loaded to the repository, it can be used in time expressions throughout Flux. You can use this Business Interval in any Timer Trigger as shown below:

If you have a Timer Trigger where you would like to apply your business interval, you will need to include some special characters in your time expression. These characters are:

  • b
  • h

In a Cron-style time expression, the "b" character indicates all values for the given column that are included in the business interval, while the "h" character indicates all values that are excluded. For example, the Cron-style expression 0 0 0 *b indicates "fire at the top of the hour for every hour value included in the interval".

All the information along with more usages of Cron expressions and business intervals can be found in Business Intervals.

Download the above workflow with the Null Action already preconfigured with the Prescript here: FLUX-SAMPLES-Create-Business-Calendars.ffc

SFTP File Transfer Workflow (Advanced)

A common pattern encountered by workflow designers involves a workflow that fires at a given time every weekday, then polls a remote server for files which are fetched and processed. Often such workflows need to support selecting files for processing based on a naming convention – such as a file description followed by a valid date in the filename. 
A best practice in Flux workflow design is accommodating for exceptional situations during processing. Often customers utilize wild card processing to select and process sets of files, not realizing that if a file in the set fails to correctly process, all following files in the set will fail to process. 

The workflow below starts with a timer trigger to fire at specified intervals. A File Exists Trigger fires collecting the set of filenames available to fetch from the remote server. Then, using a For Each Collection Element Action that submits each filename to a Regular Expression Action to compare a substring of the file to match a valid date (in yyyy-MM-dd format). If a filename matches the regular expression, the filename is submitted to a File Move Action where the file itself is moved to a local folder to be decrypted. If a filename does not match the regular expression, then the flow goes back to the For Each Collection Action to pick up the next filename in line.  
After fetching the remote files, the workflow uses another file trigger to pick up the filenames of local files to be decrypted. This pattern of using a For Each Collection Element Action provides per-file processing, meaning (in this case) if the decryption action fails, the Flux Operations Console has access to the filename of the file that failed (rather than a list of files and no knowledge of which file failed to decrypt). This same pattern is used again to re-encrypt the files.  
If anything should fail in the workflow, mail actions send an email to a set of specified email addresses to alert the customer's team that something failed. After the email has been sent (containing the filenames which failed) the normal processing of the remaining files continues. 

 

Runtime Configuration

Directories, server names, and numerous other elements of this workflow are externalized to runtime configuration properties allowing the same workflow to be used as a template for many purposes, providing consistency of deployment. This example assumes that a runtime configuration file has been defined as follows:

 

Runtime Configuration Properties  Expand source

 

Download Here: FLUX-SAMPLES-MFT-Example.ffc

Monitor Deadlines and SLAs (Advanced)

A very common scenario is the need to meet SLAs (or Service Level Agreements). Flux offers a robust way of dealing with SLAs: you can set deadlines on individual workflows (as shown below), so that if a workflow exceeds its deadline, it will publish an audit trail event indicating that the deadline has passed. The Operations Console will also graphically indicate any workflows that are approaching (or have exceeded) their deadlines.

The Deadline Time expression species the time frame that the workflow (or workflow run) is expected to be completed in. For example, a deadline time expression of "+7m" means the workflow (or each run in the workflow) is expected to finish within seven minutes.

The Deadline Window specifies how soon before the deadline the engine will publish the event flux.audittrail.server.DeadlineApproachingEvent to the audit trail. This event allows you to use the audit trail to view any workflows that are approaching their deadline. For example, a time expression of "-7m" would mean "seven minutes before the date and time of the deadline.

In this example, we have a pair of workflows – one that polls for files, and another in charge of monitoring the SLAs of the first one. 

The first workflow is watching a folder for files, we've pointed it to a folder that contains no files for the sake of the example: we want the workflow to exceed the deadline. 

The second workflow is watching for the Audit Trail events flux.audittrail.server.DeadlineApproachingEvent and flux.audittrail.server.DeadlineExceededEvent. It will print a message when each of those messages are published to the Audit Trail. 

Download here: FileWatcher with deadline.ffcSLA Monitor.ffc

Delete Files Older than 15 Days (Advanced)

This workflow polls a folder for files, calculates how long it's been since they were last updated/modified, and deletes all files that have a modification date of 15 or more days ago. 

It uses Runtime Data Mapping to pass on the File Exist trigger's RESULT.fileinfo_matches to a Collection, and to select the lastModified and url of each file to flow context variables to be used in the Null Action. The Null Action has a script in place that takes care of calculating how many days have passed since each file was modified, like so:

Then, if 15 or more days have passed since the file was modified, the workflow flows into a File Delete action, which deletes the file and flow back to the For Each Collection Element action to process the next file in line. 

Download here: FLUX-SAMPLES-RemoveOldFiles.ffc

Query a DataBase, Build a Collection with the Results and Iterate through that collection to Query a 2nd Database (Advanced)

This example demonstrates how to query a database, build a collection with the results and then iterate through the collection using the data to query a second database (or database table).

Why Can't I Run A Query Inside of Another Query?

Icon

Flux has a number of triggers and actions that provide the ability to iterate over their results within a flow within a workflow. These include the Database Query Action, Database Stored Procedure Action, and Regular Expression Action. In order to iterate over the results of these actions, a looping flow has to be defined to process all the results from the action or trigger.

Note that if, for instance, a database action tries to executed another database query action within a looping flow, a transaction break with the start of the 2nd database query action, closing the database result set from the first query. This result in a Flux exception and the workflow will fail.

This is a feature of Flux in its transaction management to provide for failover and load balancing. Since the transaction is committed, resources (including the result set from the database action) are closed. Database Result Sets, since they are operating system resources, cannot be serialized and saved to the Flux database and restored at a later point in the case of system failover or a restart. Therefore, the contents of a database result set need to be accumulated into an intermediate collection that CAN be serialized and stored for recovery.

This example illustrates how to avoid this issue and still perform queries within queries while abiding by Flux's transaction management.

 

This example uses the built-in database connection to Flux's internal database, and queries the Flux Ready Table. If you are querying a database other than Flux's, make sure you uncheck the checkbox "Built-In Database Connection Used."

The Start Action has a postscript added that creates a new List and saves it to the flowContext to be used later:

We query the database and map the RESULT.row to a Flow Context variable using Runtime Data Mapping (check the flow properties of the flow between the DB Query and the Console action). Using a postscript on the Console action 'Accumulate Query Results Into List', we fill the List we created with the values retrieved from the database:

Once all resulting rows from the DB query are exhausted, we map the List to a For Each Collection Element action's Collection property. This is done, again, using Runtime Data Mapping; only this time it's in the ELSE flow (orange) between the DB Query for Pending Sends and the FECEA.

After that, the workflow uses the collection of data from the FECEA to query another database/table/etc. In this example we are simply querying the Flux Ready table again, using the primary keys collected in rows from the first query. When all the items in the collection have been exhausted, the workflow prints out a Finished message and exits.

Download Here: FLUX-SAMPLES-QueryDBIterateThroughResultAndQuery2DB.ffc

Query a DataBase, Build a Collection with the Results and Initiate Multiple Workflows (Advanced)

A variation of the workflow shown above is to use a database query to start other workflows. 

See the parent workflows here: FLUX-SAMPLES-InitiateWorkflowsFromDatabaseQuery.ffc initiating this simple child workflow (asynchronously) here: DumpVariablesToConsole.ffc

 

Factory factory = Factory.makeInstance();

Engine engine = factory.lookupEngine("localhost", 7520);

engine.login("admin", "admin");

HashMap map = new HashMap();

for (int i = 1; i < 500; i++) {

  map.put("Message", i);

  engine.putFromRepository("/OutputMessage", "/Test" + i, map, false);

  }

engine.logout();

System.out.println("Finished outputting");

  • No labels