Skip to end of metadata
Go to start of metadata

The pages in this section discuss specific details related to the individual file triggers and actions, including the file move action, file exist trigger, file copy action, and more.

The following concepts and properties are shared across all file triggers and actions. If you are looking for more specific information about a particular type of file trigger or action, refer to the page for that trigger or action.

System Clocks with Remote Servers

If you are using a file trigger or action with a remote server (UNC, FTP, FTPS, or SFTP), take care to ensure that the system clock on the machine where your engine is running matches the clock on the remote machine.

This is especially important if you are using a stable period with a file trigger, or in general if you are performing any operation using the timestamp of the file. Timestamps are compared against the local date (the date and time from the system clock on the engine server), so if the local date and time do not match the date and time from the remote server, any operations involving timestamps are likely to be compromised.

In short, it is vital that the date, time, and time zone are matched from the engine server to the remote server in order to ensure the integrity of any time-based calculations. We recommend using a Network Time Protocol (NTP) to synchronized system clocks across multiple servers.

File Criteria

Most file triggers and actions can use a File Criteria. The File Criteria specifies what files will participate in file triggers and actions. They also specify the hosts where these files reside. These hosts can be the local host or any remote host.

Include

File Criteria includes specify the pattern that Flux should use to locate files on the system. Includes should contain both path and filename (or wildcard file pattern) information.

For example:

/path/to/myfile.txt

This include will locate any file with the literal name and extension myfile.txt that is locate in the folder /path/to. Any file matching this name and location is included in the file operation.

You can also use the "*" character to include any files that match a particular pattern:

/path/to/*.txt

This include will allow all files with the extension .txt to be included in the file operation.

Similarly, you can simply set the include to "*" to include every file in the directory.

/path/to/*

You can also use a wildcard in the middle of a pattern:

/path/to/a*z.txt

This will include any .txt file that begins with the letter "a" and ends with the letter "z".

Flux also supports another wildcard character, "?". Unlike "*" (which matches any number of characters), the "?" will only match a single character.

For example, the include:

/path/to/?.txt

Will match any .txt file whose name contains just a single character. This means the include would match a.txt, but not aa.txt.

Sometimes, you'll want to extend your include to files not just in a directory, but also any sub-directories under the directory you're polling. There are two ways to do this in a file include.

First, you can use the special syntax "*/", which indicates that Flux should check the directory being polled and any sub-directories that are exactly one level below it. This means that the include:

/path/to/*/*.txt

Will match any .txt files in the /path/to/ directory, and any sub-directories exactly one level below the directory '/path/to/'.

You can also mix-and-match this with other include concepts – for example, to get all .txt files that end with the character "z" that are in the given directory and sub-directories one level lower than the base, you would use the include:

/path/to/*/*z.txt

You can also match all sub-directories lower than that, even if they are more than one level down. To do this, use the special syntax "**/". For example:

/path/to/**/*.txt

This will match all .txt files in the given directory '/path/to/' and all sub-directories that are at any level below it.

In Flux File Criterias, the forward slash ("/") and back slash ("\") characters can be used interchangeably on both Windows and Unix systems. Flux is capable of automatically converting these characters internally to match your file system, so you do not need to worry about adjusting your File Criteria to match the particular file system it will operate on.

The Following Table lists all of the acceptable file criteria formatters and wildcards and their meanings:

Symbol

Meaning

*

Include all file names in the current directory. Directory names are not included. File and directory names in any sub-directories are not included.

**

Include all file and directory names off the current directory and in all sub-directories indirectly reachable from the current directory.

*.txt

Includes all file names ending with the literal ".txt" in the current directory. No files in any sub-directories are included.

?.txt

Includes all file names ending with the literal ".txt" in the current directory that have exactly one character in front of the ".txt" suffix. No files in any sub-directories are included.

*/*.txt

Includes all file names ending with the literal ".txt" in each sub-directory directly beneath the current directory. No deeper sub-directories are included. Does not include any ".txt" files in the current directory.

**/*.txt

Includes all file names ending with the literal ".txt" in the current directory and in all sub-directories.

myfile

Includes the literal name myfile. If the literal name myfile already exists as a file or directory, myfile refers to that file or directory. If myfile does not exist, myfile refers to a file, not a directory.

Whenever a trailing slash is used in the context of searching for files, "**" is silently appended to the symbol. For example, if "mydirectory/" is specified, then "mydirectory/" is silently translated to "mydirectory/**", which means that all files indirectly reachable from "mydirectory" are included.

If multiple includes are specified on a file criteria, Flux will use "OR" logic to match the criteria. For example, if a file criteria contains the following two includes:

a.txt
b.txt

The file criteria will match if either a.txt OR b.txt exists.

To emulate "AND" logic with file criteria, you can place each include on a separate action or trigger, then have the actions / triggers flow into a join point that waits for all to complete.

Minimum Count

The minimum count setting specifies a required minimum number of matching files to be found when using wildcard file patterns. The file criteria is satisfied when the number of files matching the file pattern reaches the specified minimum.

For most file triggers, the trigger will fire if the total number of matching files is greater than or equal to the minimum count. The File Not Exist Trigger, however, will fire if the total number of matching files is less than the minimum count.

For file actions, if the action cannot locate enough source files to satisfy the minimum count for this include, the action will throw a flux.file.FileActionException and the flow chart will enter the standard error handling mechanism.

Starting with Build 2367

Note - minimum counts can now be negative, as well as 0 and greater. See below for the associated behavior.

  1. If one or more include patterns include a minimum count >=1, all such patterns must be satisfied to continue

  2. If one or more include patterns include a minimum count =0, that pattern is optional and will return any files found, and if no files are found it will return an empty result. This pattern can be used to make the routing of processing dependent upon the presence or absence of files

  3. If two or more include patterns include a minimum count <= -1, one of files requested by these “negative patterns” must be satisfied to continue. If the action only contains a single negative pattern, it will be treated as a minimum count of 0 (see 2 above).

  4. The above can be mixed and matched in the same action.


Using a Runtime Data Map to pass the RESULT from a File Trigger to the Source of a File Action

When a File Exist trigger executes, you can tell Flux to take the list of files it retrieves and pass it on to the source of a File Action (File Copy, File Move, etc.) for processing. This can easily be accomplished using Runtime Data Mapping. 

Double-click on a flow that connects the File Exist Trigger and the File Action, and add a new Runtime Data Map by clicking on the '+' button next to 'Runtime Data Map':

A new popup window will appear. In this new window you need to set the Runtime Data Map's source and targets. You'll need to map the 'RESULT.url_string_matches' source to the target property 'File Criteria Sources':

Save all after editing and you'll be good to go. 

Exclude

To exclude files from the group of files that will participate in a file trigger or action, call FileCriteria.exclude(). This method accepts a file pattern of files to exclude. This file pattern follows the same rules as defined in the above section. All the included file patterns are processed first, followed by the excluded file patterns.

Suppose that you have three text files in your /path/to/ directory: "a.txt", "b.txt", and "c.txt". Now suppose you want to include all text files except "b.txt". The following exclude would allow "a.txt" and "c.txt" to be picked up, while ignoring "b.txt":

/path/to/b.txt

Like includes, excludes must contain both path and filename information.

Regex Filter

You can also exclude files by specifying a regular expression filter. Regular expressions are powerful, compact expressions that are used to match strings against patterns. A Regex Filter is simply a regular expression that excludes matching files from the group of files that will participate in a file trigger or action.

Further information about regular expressions can be found online:

http://java.sun.com/docs/books/tutorial/essential/regex/

For example:fileCriteria.include("*.txt");
fileCriteria.addRegexFilter(".*a\.txt");

The above method calls first include all text files in the current directory. Then any file ending with "a.txt" is filtered out, using the regular expression ".a\.txt". Note that the regular expression filter is matched against the full path name of each file. Consequently, a leading "." is needed in front of "a\.txt" to match against, for example, c:\temp\a.txt.

If you'd like to test your regex before using them in Flux, you can use http://www.regexpal.com/ .

File Copy and File Move Actions

File Copy and File Move Actions have some additional options available when using a File Criteria. Since these options are only used when transferring a file to a new location, they are not available on other action types (like the File Delete Action) that do not transfer files during their operation.

Renamer

Renamers allow you to change the name of a file at the target location during a copy or move operation. There are two renamers available:

Global Renamer

The Global Renamer is used to rename files matching a particular pattern using simple wildcard expressions. The wildcard character, '*', can be used to match zero or more characters within a filename.

Typically Global Renamers are used for changing the extension of a file or group of files. For example, to convert all .java files to .txt, you could use a Global Renamer with the following settings:

From File Pattern

*.received

To File Pattern

*.validated

The renamer "From File Pattern" and "To File Pattern" must contain one wildcard '*' each. The characters matched in the From Pattern will replace the "*" character in the To Pattern when the file action runs (for example, if the From Pattern is "*.zip" and the To Pattern ".txt", and the action matched a file "my.zip", the file would be renamed to "my.txt" when the action executed).

Regular Expression Renamer Examples

The Regular Expression Renamer is used to rename files matching a particular regular expression pattern. For example, to modify all filenames ending with abc.txt and make them end instead with xyz.txt, you could use the following settings:

From File Pattern

(.*)abc\.txt

(?i)(Budget_.*).(.*) e.g., for a file named BUDGET_Capital.xlsx

To File Pattern

\1xyz.txt

${date yyyy-MM-dd}_\1_Reviewed.\2 e.g., to a file named 2017-09-22_Budget_Capital_Reviewed.xlsx

The pattern "\<number>" syntax can be used to access a matched portion from the regular expression in the From File Pattern (represented by "(.*)"). "\1" retrieves the first matched portion, "\2" the second, and so on.

For more information on regular expression usage, see Oracle's guide to regular expressions.

Preserve Directory Structure

The Preserve Directory Structure flag indicates whether Flux should attempt to recreate the directory structure leading to a file after it is moved or copied.

In order to use this flag, the include pattern that matches a file must contain the sub-directory matching pattern, "/**/".

When this flag is enabled, a new directory structure is taken on the target, beginning from the directory that precedes the '/**/'. For example, for a file criteria of '/path/to/**/*.txt', the new directory structure will be created starting from /path/to/ of the include and ending with the directory where the source file actually resides.

For example, consider a File Criteria with a source criteria of:

/home/flux/**/*.*

That matches a file named:

/home/flux/path/to/myfile.txt

And has a target Directory of:

/mytargetfolder/

When Preserve Directory Structure is enabled, then the final path of the file on the target will be:

/mytargetfolder/path/to/myfile.txt

On the other hand, if Preserve Directory Structure is disabled, the final path of the file on the target will be:

/mytargetfolder/myfile.txt

If you enable Preserve Directory Structure, but are not using the sub-directory search '/**/'; Flux will reproduce the entire path from the root. So, for an include like '/home/flux/path/*/*.png' the directory tree that will be created in the target folder will be '/home/flux/path/<matching-sub-dir>/'.

Network Hosts

FTP Hosts

The files and directories specified in a File Criteria object normally refer to files and directories on the local file system. However, you can specify that these files are located on a remote FTP server. This functionality allows you to specify files and directories on a remote FTP server that should be used by the different file triggers and actions.

For example, by setting an FTP host, you can copy files to and from an FTP server. You can also have a File Trigger fire when a file on an FTP server changes its state.

To create and initialize an FTP host, first create an FtpHost object from a FileFactory object. The FtpHost object contains methods for setting the name of the FTP host, the port number, and username and password information for logging in. Finally, the FTP host is set by calling:fileCriteria.setHost(ftpHost);

After calling this method, the File Criteria’s included files and directories are relative to the default landing directory on this FTP host for the FTP user. Furthermore, calling fileCriteria.setHost() resets that landing directory to the default root directory for FTP servers, "/".

If your FTP server does not display directory listings in the standard format, you can substitute your own parser to adapt non-standard FTP directory listings to the scheduler. You need to create a class with a default constructor that implements the flux.file.FtpFileListParser interface and provide that class to your FtpHost object using the setFileListParser() method.

Non-standard FTP Servers

Some FTP servers, such as special-purpose FTP servers used for EDI purposes, return non-standard file listings when responding to FTP commands. Flux supports these non-standard FTP servers by allowing the workflow to supply its own Java class that is responsible for parsing the non-standard file listings. Thus, Flux can monitor and manipulate files that reside on non-standard FTP servers.

Use the FtpHost.setFileListParser() method to register your custom file listing parser. This class must implement the flux.file.FtpFileListParser interface.

Secure FTP Host

A Secure FTP Host is exactly like an FTP Host, except that it operates on secure FTP servers instead of standard FTP servers. Secure FTP servers encrypt the communications traffic for security purposes.

When using Secure FTP (SFTP) with the Flux Operations Console, workflows and error messages may have problems being displayed inside of the web application due to classpath issues. To prevent this problem from occurring either place the flux.jar file inside your invoking JVM's lib/ext directory, or include flux.jar in the system classpath.

Using a Username / Password with a Private Key

When using a Secure FTP Host, you can set a username / password while using a private key. In the editor, just set the username / password as normal, then click the "Set Private Key" button to set the private key. Your username and password credentials will be saved in addition to the key settings.

FTP Over SSL

FTP over SSL (FTPS) encompasses both FTP and SFTP functionality. FTPS provides a way to perform secure file transfers using an SSL/TLS layer. When connecting to FTPS servers using Flux, the connection may take longer than usual to initialize depending upon which type of connection strategy the FTPS server is utilizing. The possibility of an extended connection wait is due to a series of connection strategies tried until the proper strategy is found. Connection via the TLS layer is first attempted, the second attempt is a connection over SSL, and lastly an attempt is made to connect via implicit SSL. Each attempt is given five seconds to connect. If the connection can not be made within the time frame, the next connection strategy is attempted.

User NameUser PasswordServer Certificate FilenameServer PasswordComment
Valid UsernameValid Password No FilenameNo Password 

If FTP Server is configured for certificate, transfer will fail with: Check your username, password, certificates, and that you are using the proper authentication type for the server: unexpected_message(10)

If FTP Server is not configured for certificate, will connect and transfer.

Valid UsernameValid Password Valid Filename for certificateValid Password for certificate store

If FTP Server is configured for certificate, will connect and transfer.

If FTP Server not configured for certificate, transfer will fail with: Check your username, password, certificates, and that you are using the proper authentication type for the server: certificate_unknown(46)

Invalid UsernameAny or No Password Any or No FilenameAny or No Password Transfer will fail with: Check your username, password, certificates, and that you are using the proper authentication type for the server: unexpected_message(10)
Valid UsernameInvalid Password or No PasswordAny or No FilenameAny or No Password Transfer will fail with: Check your username, password, certificates, and that you are using the proper authentication type for the server: unexpected_message(10)
Valid UsernameValid Password Valid Filename for a certificate, but not the correct certificate for the FTP ServerValid Password for certificate store

If FTP Server is configured for certificate, transfer will fail with: Check your username, password, certificates, and that you are using the proper authentication type for the server: unexpected_message(10)

If FTP Server not configured for certificate, transfer will fail with: Check your username, password, certificates, and that you are using the proper authentication type for the server: certificate_unknown(46)

Valid UsernameValid Password Valid Filename for a certificate, but not the correct certificate for the FTP ServerInvalid Password for certificate store or No Entry

If FTP Server is configured for certificate, transfer will fail with: Check your username, password, certificates, and that you are using the proper authentication type for the server: Keystore was tampered with, or password was incorrect

If FTP Server is not configured for certificate, transfer will fail with: Check your username, password, certificates, and that you are using the proper authentication type for the server: Keystore was tampered with, or password was incorrect

Valid UsernameValid Password Valid Filename for certificateInvalid Password for certificate store or No Entry

If FTP Server is configured for certificate, transfer will fail with: Keystore was tampered with, or password was incorrect.

If FTP Server not configured for certificate, transfer will fail with: Check your username, password, certificates, and that you are using the proper authentication type for the server: certificate_unknown(46)

Valid UsernameValid Password Invalid FilenameAny or No Password 

If FTP Server is configured for certificate, transfer will fail with: Check your username, password, certificates, and that you are using the proper authentication type for the server: <certificate filename >

If FTP Server is not configured for certificate, transfer will fail with: Check your username, password, certificates, and that you are using the proper authentication type for the server: <certificate filename >

Valid UsernameValid Password Valid Filename but not a certificate fileAny or No Password 

If FTP Server is configured for certificate, transfer will fail with: Check your username, password, certificates, and that you are using the proper authentication type for the server: Invalid keystore format

If FTP Server is not configured for certificate, transfer will fail with: Check your username, password, certificates, and that you are using the proper authentication type for the server: Invalid keystore format

FTPS Certificates and Key Store

A Keystore that is defined server side is there to provide certs to be validated by connecting clients if the client is asking for a certificate to confirm with. 

In this scenario, it is the client's responsibility to define whether or not it wants to confirm the server's certificate.

If the client connecting provides no certificate to compare, the server will only authenticate with the username and password provided. 

In Flux, the decision to accept the certificate provided to the client by the server is done by checking if a certificate file was defined in the Action, if it was, it will compare the client side certificate with the server side cert to see if they are the same. If they aren't it will fail to authenticate.

If no certificate file is defined in the Flux Action, it will automatically accept whatever cert the server provides. 


The below is caused by a server not being able to read the key provided to it in the defined keystore because no Key Password was defined. This will cause authentication failure and close the connection.


Supported FTP Servers in Flux

Flux explicitly supports two File Transfer Protocol (FTP) servers, although other FTP servers are known to work with Flux. These servers are listed below.

  • Microsoft's Internet Information Services (IIS) FTP server version 5.0+
  • Very Secure FTP server (VSFTP) 2.0+

Although the above servers have been thoroughly tested against Flux and are fully supported, other FTP servers have been known to work correctly. If you are using any other FTP servers other than what is listed above, note that they have not been tested and are not officially supported by Flux.

IIS 5.0 NOTE: When using IIS 5.0, a problem occurs when changing into directories with spaces in their names. A workaround is to either enable the "Use CD Commands" option in the Flux Designer under the FTP Host section within the "File Criteria" editor, or set the "issueCdCommandsMode" flag to true on a flux.file.FtpHost object in Java code.

Passive Mode

Passive network connections are firewall-friendly but require support from the FTP server. The passive mode option is available on FTP, Secure FTP, and FTP over SSL hosts. Connections established with these hosts are passive by default. If passive mode is disabled, active mode will be used.

Issue Cd Commands

The Issue Cd Commands option specifies how Flux changes directories when using an FTP, Secure FTP, or FTP over SSL host. If issue cd commands is enabled, Flux will issue a CD command to the server for each directory. If the option is disabled, Flux will attempt to issue a single CD command for the entire path. This option is useful for decreasing the number of commands Flux issues to your server. Issue CD commands is disabled by default.

NOTE: Certain FTP servers do not support specifying an entire path with the CD command.

Binary/ASCII Transfer Mode

File transfers with Flux can be done in two modes, Binary or ASCII. Binary mode is for byte-oriented files while ASCII mode is used for text-oriented files. This option can be set on an FTP, Secure FTP, or FTP over SSL host and will be used for all files transferred by that host. Binary transfer mode is enabled by default.

Universal Naming Convention (UNC)

Flux also supports connection to network devices via UNC. When using a UNC host in Flux, the user may specify an optional domain, username, password, and port. The name of the server and file being used are required fields.

Specify a Default Username, Password, and Domain for UNC Hosts

You can specify default username, password and domain settings by setting the following JVM options on the JVM where the Flux engine runs:

-Djcifs.smb.client.username=<username> -Djcifs.smb.client.password=<password> -Djcifs.smb.client.domain=<domain>

Where <username>, <password>, and <domain> are replaced (respectively) with the username, password and domain settings for your UNC server.

Once these properties are in place, any file trigger or action that uses a UNC host and does not have a username, password, or domain specified will use the settings defined above.

A value set directly on the trigger or action will override the default value, allowing you to configure specific values for particular actions while retaining the default settings for other actions in your workflows.

These settings are applied system-wide and are not dependent on workflow namespace.

Specifying Defaults when Flux Runs as a Windows Service

You can specify these default settings by adding the following lines to the engine.conf file for your Windows service:

wrapper.java.additional.13=-Djcifs.smb.client.username=<username>
wrapper.java.additional.14=-Djcifs.smb.client.password=<password>
wrapper.java.additional.15=-Djcifs.smb.client.domain=<domain>

Zip File Criteria

A Zip File Criteria is the same as a File Criteria, except that the included and excluded files are relative to the files contained in a given Zip file. The root directory is also relative to the root of the Zip file itself.

After creating a ZipFileCriteria object, you must specify the actual name of the Zip file by calling ZipFileCriteria.setZipFilename().

For example, suppose a Zip file a.zip contains the files /myfile.txt and /mydir/myOtherFile.txt. You can specify these two text files in a Zip File Criteria as follows.

zipFileCriteria.setZipFilename("a.zip");
zipFileCriteria.include("*/.txt");

These two method calls include the text files /myfile.txt and /mydir/myOtherFile.txt from inside the Zip file for use in a file trigger or action.

Zip File Criteria objects can be used for creating Zip files, copying files into existing Zip files, and extracting files from Zip files.

File Actions and Transaction Breaks

If you are using file actions (including the file copy, move, delete, and create actions) within a workflow, and the workflow fails or must be rolled back to the last transaction break for any reason, it is important to note that any work performed on the file system will not be rolled back along with the action itself.

This means that, although the state and execution of the workflow are rolled back in the database as normal, any file operations — meaning any file that was deleted, moved, copied, or created — are retained on the file system. If and when the workflow begins re-executing, the file action will simply perform its operation again, overwriting any files (or partial data) that was written to the system on the previous rolled-back attempt.

If, instead, you need the file system to be cleaned up or "reset" to a different state, you will need to design an error handler or error flow that performs the file cleanup. For example, if your workflow contains a file move action that might be rolled back, your error handler could contain a corresponding file move action to reverse the transfer in the event of an error or roll back.

File Infos

Some file triggers and actions return results that contain File Infos. This is a special data type that contains information about a file found or operated on by the trigger or action.

The file info is a complex data map, meaning it has several properties available containing specific information about the file it represents. If you have a File Info variable, you can use variable substitution to access each property (also called a "field"). For example, if a File Info is stored as the variable "fileinfo", you could use variable substitution to access the filename property: ${fileinfo.filename}.

The available properties for a File Info are:

Property Name

Description

filename

The name of this file (without path information).

lastModified

The date that the file was last modified on the file system.

parent

The system-dependant absolute path to the parent of this file (in other words, the absolute path of the directory where this file is stored).

read

A boolean (true / false) value indicating whether this file can be read.

size

The size of the file (in bytes).

url

The URL to the file.

write

A boolean (true / false) value indicating whether this file can be modified.

  • No labels