Skip to end of metadata
Go to start of metadata

The following example workflows are provided to simplify getting started with Flux. These workflows are easily modified and extended to meet your requirements.

Action prescripts and postscripts are used in these flows to provide status messaging and details of the file name being moved. See example prescript here of updating the status message in the Operations Console during a file copy: Prescript Example.png

Bulk File Copy (Basic)

This workflow scans a directory and copies all the found files. The bulk file copy is useful where all the files have to be copied at a point in time. If this bulk file copy fails, rerunning the flow will copy all the files over again.

This example assumes that a runtime configuration file has been defined, and that runtime configuration properties to identify the file source directory and target directory specified as follows:

 

Required Runtime Configuration File Properties

Download here: Bulk-Copy.ffc

 

Mass File Copy and Prepend Today's Date to Each Copied File (Basic)

This workflow scans a directory and copies all the found files. The workflow consists of just a single File Copy Action. A Regular Expression Renamer is configured to prepend the date to each copied file.

Note in the illustration below that the From Pattern is (.*). This regular expression captures the file name into a group, later referenced in the To Pattern as \1 (Since it is the first and only regex group captured).

The To Pattern is ${$date yyyy-MM-dd}-\1. This pattern prepends the date in a form such as 2014-Jun-18, with a dash '-', in front of each file name copied from the current directory into c:\test.

                               

Download here: MassFileCopyWithRegexToPrependDate.ffc

Generic File Move, Copy, Rename, Delete (Basic)

This workflow creates a file, tests for its existence, moves it somewhere, copies it back to where it was moved from, renames it, and deletes both files and tests to make sure they both no longer exist.

This example assumes that a runtime configuration file has been defined, and that runtime configuration properties to identify the file source directory and target directory specified as follows:

 

Required Runtime Configuration File Properties

Download here: FLUX-SAMPLES-FileActions.ffc

Read a CSV File and Print Result (Basic)

This example uses the CsvJdbc library to connect to a CSV file located in the local system, and perform a query. 

This example assumes that you have the 'csvjdbc-1.0-20.jar' in your Flux Classpath; and a CSV file named 'customers.csv' in the path '/Users/Documents'. Using Flux's built-in Database Query action it creates a connection to the location of the file and then performs the query set in the 'Query' property of the Database Query action. 

The JDBC URL property specifies the path to the file, to change the path to point to another location, just append the path to 'jdbc:relique:csv:' as shown below:

In the Query we specify the file name (without file extension):

Where 'customers' is the name of our CSV file. The output should look like:

Download here: FLUX-SAMPLES-CSVJDBC.ffccustomers.csvcsvjdbc-1.0-20.jar

Ping a Host and Notify if Down (Basic)

This workflow sends a ping request to a host and email a notification if the host is down.

This example assumes that a shell script has been defined in a directory located at /home/flux/FluxTest/Ping. Modify the Process Action to point to a directory of your choosing or enter the full path to the script or executable in the Command text field. Paths can be relative to the directory was Flux was installed or an absolute path can be used (as in this example).

You will also have to modify the Mail Action to input the correct mail server settings like so:

 

Required Shell Script for Linux/Unix
Required Batch File for Windows

 

Note: You'll need to check that your antivirus software doesn't have any rules blocking SMTP. For example, McAfee Enterprise Virus protection - standard for many corporations - has a default setting that blocks SMTP ports 25 and 587. The default rule needs to be modified to add java.exe and telnet.exe as programs allowed to use the mail ports. This is illustrated in the attached screenshot.


 

Download here: FLUX-SAMPLES-Ping.ffc

Mail Trigger that prints to STDOUT the Subject and Body of the email (Basic)

This workflow polls the 'Inbox' of a test mail user and fires when new mail arrives; it then prints out to STDOUT the email subject and the email body. Export the workflow to the engine and then send an email to 'fluxtemplate@gmail.com' (without quotes) and check the Console Action's output match the subject and body. 

Download here: FLUX-SAMPLES-MailTrigger.ffc

Flow Chart Trigger that waits for another workflow to finish copying files and then iterates through them and prints to STDOUT (Basic)

This workflow waits on a Flow Chart trigger until the workflow defined in the 'Namespace' property finishes and then polls a folder and iterates through the list of files found. The 'Namespace' can also be used to define entire namespaces, specific workflows, etc. as explained in Flow Chart Trigger.

You will need to edit the File Exist trigger to point to a valid directory in your system. 

This example waits for the /Incoming/Daily workflow attached to finish. You will need to change the 'Source' and 'Target' of the File Copy action to valid directories in your system. 

Download here: Daily-FlowChartTrigger.ffcIncoming-Daily.ffc

Using a For Each Collection Element action to iterate through a collection (Basic)

This workflow scans a folder for files, iterates through them and prints out the filenames one at a time.

It uses a File Exist trigger to poll for files, then Runtime Data Mapping to pass the resulting collection to a For Each Collection Element action like so:

We're calling ${i} from the 'Print Filenames' Console action, 'i' is the Loop Index of the For Each Collection Element action, and we can use Variable Substitution to call that variable from anywhere in the workflow (keep in mind that until the For Each Collection Element action runs, that variable is empty). 

You will need to modify the File Exist trigger's source folder to a valid path in your environment. 

Download here: FLUX-SAMPLES-IterateFiles.ffc

Calling a workflow using a FlowChart action (Basic)

This workflow exports a workflow that's in the Workflow Repository under a new name, waits for it to finish, and then continues executing. 

To run this example you'll need to download both workflows and upload them to your Repository. Then you can start the /FLUX-SAMPLES/FlowChartAction workflow which will in turn start the child workflow. 

Download here: FLUX-SAMPLES-FlowChartAction.ffcFLUX-SAMPLES-ChildWorkflow.ffc

File Trigger that Times Out and Sends Notification Email (Basic)

This workflow scans two different folders for files and then copies them to the same location. The File Exist triggers have a timeout – the triggers will poll the source folders and if nothing is found after 15 minutes, the triggers time out and exit. When the triggers time out, they flow into an Email action, which sends a notification email to let you know that the files were not found in the source folder as expected. 

To run this workflow, the sources for the File Exist triggers, the targets for the File Copy actions and the mail user, password and SMTP server settings of the Mail actions need to be updated to suit the environment where the workflow will be run. Sample values are provided for all, to make it easier to understand and change them.

Download here: FLUX-SAMPLES-FileTriggerWithTimeout.ffc

File Copy that Fails and Exits on Error (Intermediate)

This workflow scans a directory and copies a file at a time. If a file copy fails, the workflow is set to ‘FAILED’ in the Operations Console. From the console, you can either ‘restart’ the workflow to run from the beginning, or ‘recover’ the workflow which starts the flow at the point where the file copy failed.

This example assumes that a runtime configuration file has been defined, and that runtime configuration properties to identify the file source directory and target directory specified as follows:

 

Required Runtime Configuration File Properties

Download here: FLUX-SAMPLES-FileCopy-ExitOnError.ffc

Invoking a Rest Service and Print Result to STDOUT  (Intermediate)

This example illustrates how to invoke a REST service on a remote system using Flux. The Rest Action has only one required property: baseURL, which specifies the location of the HTTP service to invoke. Arguments can be passed to the REST service and the REST service can also return values which can be used in the flow chart.

The above screenshot illustrates a Rest Action which invokes the Yahoo StockQuote service at http://finance.yahoo.com/d/quotes.csv. The service takes two arguments: 1. quotes string argument, “GOOG+YHOO+HPQ+IBM+MSFT+ORCL”, and  2. format string argument "snl1d1t1ohgdr", which are passed to the REST service.

The result returned from the REST service is of type RestAction.RestActionResult. The Rest Action postscript maps result.result to a flow context variable "quote".

Variable substitution is used in the Console Action to display the value of the quote flow context variable; and will display the output on the STDOUT.

Download here: FLUX-SAMPLES-GetStockQuote.ffc

File Copy that Continues on Error (Intermediate)

This workflow scans a directory and copies a file at a time and logs a notification in the event of a file copy failure. The workflow than continues to process the next files.

 

This example assumes that a runtime configuration file has been defined, and that runtime configuration properties to identify the file source directory and target directory specified as follows:

 

Required Runtime Configuration File Properties

Download here: FLUX-SAMPLES-FileCopy-ContinueOnError.ffc

Report or Data Delivery (Intermediate)

This Flux workflow starts with a simple database table that contains the email address, file name, and a timestamp of when the file was sent (initially an empty field).  Every five minutes a timer trigger fires, triggering a database query to search the database for records where the sent timestamp is empty. For each record found, a Flux mail action builds an email using a customizable email template, and attaches the file specified in the record. The mail is sent, and the time that the mail was sent is updated in the database record. In the event of a mail failure or a file not found issue, a notification is sent.

This example assumes that a database table named 'Customers' has been defined with an email address, file path (to the file that needs to be delivered), and timestamp field (all defined as character or varchar fields). 

Download here: FLUX-SAMPLES-DB+Email.ffc

Adding and Using a Business Interval (Intermediate - Requires Scripting)

The following script code creates a business interval for use in restricting workflows to only run within specified times and dates. 

Source for Creating and Saving a Business Interval  Expand source

Create a workflow and then add a Null Action to the workflow. Take the above code and paste it into the Prescript for the Null Action and then export the workflow to the engine. Flux will run the above script and create the business intervals/calendars defined and add them into the Flux Repository, as illustrated below:

 After this business interval is loaded to the repository, it can be used in time expressions throughout Flux. You can use this Business Interval in any Timer Trigger as shown below:

If you have a Timer Trigger where you would like to apply your business interval, you will need to include some special characters in your time expression. These characters are:

  • b
  • h

In a Cron-style time expression, the "b" character indicates all values for the given column that are included in the business interval, while the "h" character indicates all values that are excluded. For example, the Cron-style expression 0 0 0 *b indicates "fire at the top of the hour for every hour value included in the interval".

All the information along with more usages of Cron expressions and business intervals can be found in Business Intervals.

Download the above workflow with the Null Action already preconfigured with the Prescript here: FLUX-SAMPLES-Create-Business-Calendars.ffc

SFTP File Transfer Workflow (Advanced)

A common pattern encountered by workflow designers involves a workflow that fires at a given time every weekday, then polls a remote server for files which are fetched and processed. Often such workflows need to support selecting files for processing based on a naming convention – such as a file description followed by a valid date in the filename. 
A best practice in Flux workflow design is accommodating for exceptional situations during processing. Often customers utilize wild card processing to select and process sets of files, not realizing that if a file in the set fails to correctly process, all following files in the set will fail to process. 

The workflow below starts with a timer trigger to fire at specified intervals. A File Exists Trigger fires collecting the set of filenames available to fetch from the remote server. Then, using a For Each Collection Element Action that submits each filename to a Regular Expression Action to compare a substring of the file to match a valid date (in yyyy-MM-dd format). If a filename matches the regular expression, the filename is submitted to a File Move Action where the file itself is moved to a local folder to be decrypted. If a filename does not match the regular expression, then the flow goes back to the For Each Collection Action to pick up the next filename in line.  
After fetching the remote files, the workflow uses another file trigger to pick up the filenames of local files to be decrypted. This pattern of using a For Each Collection Element Action provides per-file processing, meaning (in this case) if the decryption action fails, the Flux Operations Console has access to the filename of the file that failed (rather than a list of files and no knowledge of which file failed to decrypt). This same pattern is used again to re-encrypt the files.  
If anything should fail in the workflow, mail actions send an email to a set of specified email addresses to alert the customer's team that something failed. After the email has been sent (containing the filenames which failed) the normal processing of the remaining files continues. 

 

Runtime Configuration

Directories, server names, and numerous other elements of this workflow are externalized to runtime configuration properties allowing the same workflow to be used as a template for many purposes, providing consistency of deployment. This example assumes that a runtime configuration file has been defined as follows:

 

Runtime Configuration Properties  Expand source

 

Download Here: FLUX-SAMPLES-MFT-Example.ffc

Monitor Deadlines and SLAs (Advanced)

A very common scenario is the need to meet SLAs (or Service Level Agreements). Flux offers a robust way of dealing with SLAs: you can set deadlines on individual workflows (as shown below), so that if a workflow exceeds its deadline, it will publish an audit trail event indicating that the deadline has passed. The Operations Console will also graphically indicate any workflows that are approaching (or have exceeded) their deadlines.

The Deadline Time expression species the time frame that the workflow (or workflow run) is expected to be completed in. For example, a deadline time expression of "+7m" means the workflow (or each run in the workflow) is expected to finish within seven minutes.

The Deadline Window specifies how soon before the deadline the engine will publish the event flux.audittrail.server.DeadlineApproachingEvent to the audit trail. This event allows you to use the audit trail to view any workflows that are approaching their deadline. For example, a time expression of "-7m" would mean "seven minutes before the date and time of the deadline.

In this example, we have a pair of workflows – one that polls for files, and another in charge of monitoring the SLAs of the first one. 

The first workflow is watching a folder for files, we've pointed it to a folder that contains no files for the sake of the example: we want the workflow to exceed the deadline. 

The second workflow is watching for the Audit Trail events flux.audittrail.server.DeadlineApproachingEvent and flux.audittrail.server.DeadlineExceededEvent. It will print a message when each of those messages are published to the Audit Trail. 

Download here: FileWatcher with deadline.ffcSLA Monitor.ffc

Delete Files Older than 15 Days (Advanced)

This workflow polls a folder for files, calculates how long it's been since they were last updated/modified, and deletes all files that have a modification date of 15 or more days ago. 

It uses Runtime Data Mapping to pass on the File Exist trigger's RESULT.fileinfo_matches to a Collection, and to select the lastModified and url of each file to flow context variables to be used in the Null Action. The Null Action has a script in place that takes care of calculating how many days have passed since each file was modified, like so:

Then, if 15 or more days have passed since the file was modified, the workflow flows into a File Delete action, which deletes the file and flow back to the For Each Collection Element action to process the next file in line. 

Download here: FLUX-SAMPLES-RemoveOldFiles.ffc

 

  • No labels