Defining Windows Installer Custom Actions

The Windows Installer provides a large amount of built-in functionality for handling most common installation tasks. Such functionality is accessible via a host of database tables as described Windows Installer Database Reference on MSDN. Although the built-in functionality is quite extensive, there are still many cases where some custom functionality needs to be used. To support such cases, the Windows Installer provides an extensibility mechanism known as custom actions.

By and large, the Windows Installer utilizes declarative programming as the means for directing the actions of an installation. By this, I mean that the install author describes the content and desired results for an installation, rather than specifying the control flow of the steps for performing the installation. In other words, the installation is data-driven, rather than code-driven. Custom actions are a departure from this paradigm, as they provide a mechanism for an install author to invoke custom code at certain points in the installation process.

CustomAction Table

The install author describes a custom action using the CustomAction table. This table has 4-5 columns depending on the target version of the Windows Installer. The columns of this table are described below:

Column Name Description
Action Represents the name given to the custom action. It is the primary key for this table, and is the value used when referencing the custom action from other tables.
Type This is a numeric field of bit flags which specify the custom action type along with a variety of execution and return processing options.
Source Usage dependent on the custom action type
Target Usage dependent on the custom action type
ExtendedType Introduced with Windows Installer 4.5. Provides space for additional bit flags.

The Action field for a custom action must be a unique name that does not conflict with the names of any standard actions or other custom actions. The remaining fields are described in detail in the following sections.

Basic Custom Action Types

The type of a custom action is defined by a basic custom action type value plus a set of bit flags. For Windows Installer 4.0 and earlier, the custom action type is specified in the Type field of the Custom Action table. In Windows Installer 4.5, an additional field named ExtendedType was introduced to accommodate additional bit flags for the custom action type. The basic custom action type describes the implementation form of a custom action, while the bit flags define such things as how to interpret return codes from the custom action, and what points in the installation process the custom action can be scheduled to run.

Basic custom action types can be broadly classified into three primary categories:

  • Those that execute custom code
  • Those that set or change installer properties
  • Those that terminate an installation with an error

The majority of custom action types fall into the first category. These types permit the execution of compiled code in DLL and EXE files, as well as interpreted scripts in the form of JScript or VBScript.

Running a DLL Function

A custom action can call a function in a DLL file. The DLL file can either be stored in the Binary table of the MSI package, or it can be among those files that are being installed. The DLL function that is called must conform to a specific API format, as follows:

UINT __stdcall CustomAction(MSIHANDLE hInstall);

The MSIHANDLE that is automatically passed to the DLL function can be used within the function to access the current running installation, and to query or modify the attributes of the installation session.

DLL File Stored in a Binary Table Stream

The DLL file containing the function called by the custom action can be stored in the Binary table of the MSI package. This is the best place to put the DLL if it only needs to be used during the installation, or a subsequent uninstallation. When the DLL is stored in the Binary table, any custom actions which call functions in it may be scheduled at virtually any point in the installation process.

CustomAction Table Entry
Basic Custom Action Type 1 decimal (0x01 hex)
Source Column Key to the Binary table identifying the binary data stream that represents the DLL file.
Target Column The name of the DLL entry point function. This function must conform to the standard API format defined for MSI custom actions.

DLL File Installed With a Product

The custom action can also call a function in a DLL file which is installed as part of the product. This makes sense to do in cases where the DLL contains functionality that is used by the installed application, in addition to the custom action. This restricts the custom action to being scheduled at a point in the installation process where installed files are present on the target system. This is at any point following the processing of the InstallFiles standard action during a normal installation, or before the processing of the RemoveFiles standard action during an uninstallation.

CustomAction Table Entry
Basic Custom Action Type 17 decimal (0x11 hex)
Source Column Key to the File table identifying the DLL file that is installed as part of the product.
Target Column The name of the DLL entry point function. This function must conform to the standard API format defined for MSI custom actions.

Running an Executable File

Another method for executing custom code is the launch an executable file, optionally specifying command line parameters. There is not an easy way to pass a handle to the installation session to such an executable file, so EXE files are best used in cases where their actions are independent of the installation processing.

EXE File Stored in a Binary Table Stream

If the given executable file is only used by a custom action, then it is probably best to store it in the Binary table of the MSI package. This makes the EXE file available for virtually any point in the installation process, and protects it from being run directly by an end-user.

CustomAction Table Entry
Basic Custom Action Type 2 decimal (0x02 hex)
Source Column Key to the Binary table identifying the binary data stream that represents the EXE file.
Target Column Any command line arguments to pass to the executable when launching it. Since the Target column uses the Formatted data type, its value may contain references to properties, files, directories, etc.

EXE File Installed With a Product

If the custom action runs an executable file that is also used as part of the installed application, the install author can avoid having to store an extra copy in the Binary table under certain conditions. Instead, the custom action can be configured to run the executable file directly from the location where it is installed. This can only work if the custom action is called from a point in the installation process where the product files are present on the target system. Anytime after the processing of the InstallFiles standard action during a normal installation or prior to the processing of the RemoveFiles standard action during an uninstallation will work.

CustomAction Table Entry
Basic Custom Action Type 18 decimal (0x12 hex)
Source Column Key to the File table identifying the EXE file that is installed as part of the product.
Target Column Any command line arguments to pass to the executable when launching it. Since the Target column uses the Formatted data type, its value may contain references to properties, files, directories, etc.

EXE File Having a Path Referencing a Directory

For this custom action type, the working directory for running the executable can be explicitly specified. All other forms of running an executable use a default working directory.

CustomAction Table Entry
Basic Custom Action Type 34 decimal (0x22 hex)
Source Column Key to the Directory table identifying the directory to use as the working directory when running the executable.
Target Column The complete command line for launching the executable, including the full path and filename of the executable file, along with any command line options. Quotation marks must be used around long file names or paths. Since the Target column uses the Formatted data type, its value may contain references to properties, files, directories, etc.

EXE File Having a Path Specified By a Property Value

The property referenced by this custom action type must resolve to the full path and filename of the executable file to launch. A typical case for using this custom action type would be to launch an EXE file that was discovered using the searching functionality of the AppSearch table. It can also be used to launch an EXE that resides on the source installation media.

CustomAction Table Entry
Basic Custom Action Type 50 decimal (0x32 hex)
Source Column Key to the Property table identifying the property containing the full path and file name of the target executable file. The path need not be enclosed in double quotes.
Target Column Any command line arguments to pass to the executable when launching it. Since the Target column uses the Formatted data type, its value may contain references to properties, files, directories, etc.

Running a JScript Function

Custom actions can be implemented in JScript as long as the install author can guarantee that the scripting engine is available on the target system. JScript custom actions have access to the installation session via the Session object. They can use this object to query or modify the attributes of the installation session.

JScript File Stored in a Binary Table Stream

As with compiled binaries, JScript files can be stored in the Binary table of the MSI package. This option is suitable for non-trivial scripts that are only utilized during installation or uninstallation. Such scripts will be available for use at virtually any point in the installation process, and storing them in the Binary table will help to hide the scripts from end-users.

CustomAction Table Entry
Basic Custom Action Type 5 decimal (0x05 hex)
Source Column Key to the Binary table identifying the binary data stream that represents the JScript.
Target Column The name of an optional script function.

JScript File Installed With a Product

In cases where the installed product may use the same JScript file, it may make sense to avoid duplication, and run the JScript custom action using the installed file. This does restrict the points in the installation process where the custom action may be used. Mainly this includes any point after the InstallFiles standard action during a normal installation, or prior to the RemoveFiles standard action during an uninstallation.

CustomAction Table Entry
Basic Custom Action Type 21 decimal (0x15 hex)
Source Column Key to the File table identifying the JScript file that is installed as part of the product.
Target Column The name of an optional script function.

JScript Text Stored in This Sequence Table

For cases where the JScript custom action consists of a relatively small amount of script text, it may be easier to store the script directly in the definition of the custom action. This basic custom action type facilitates this feature, storing the JScript text in the Target field of the custom action. Due to field size restrictions, the JScript text is limited to a maximum length of 255 characters.

CustomAction Table Entry
Basic Custom Action Type 37 decimal (0x25 hex)
Source Column null
Target Column Literal JScript text

JScript Text Specified by a Property Value

An alternative method for storing the JScript text for a custom action is to place it in an entry in the Property table. That property can then be referenced by the custom action. Property values can accommodate longer text than the Target field of the CustomAction table, so longer scripts can be used in this case. However, for maintenance purposes, it is generally best to store relatively short scripts in the Property table, and to use the Binary table for longer scripts.

CustomAction Table Entry
Basic Custom Action Type 53 decimal (0x35 hex)
Source Column Key to the Property table identifying the property containing the script text.
Target Column The name of an optional script function.

Running a VBScript Function

As with JScript, custom actions can be implemented in VBScript as long as the install author can guarantee that the scripting engine is available on the target system. VBScript custom actions also have access to the installation session via the Session object, which can be used to query or modify the attributes of the installation session.

VBScript File Stored in a Binary Table Stream

VBScript files can be stored in the Binary table of the MSI package. This option is suitable for non-trivial scripts that are only utilized during installation or uninstallation. Such scripts will be available for use at virtually any point in the installation process, and storing them in the Binary table will help to hide the scripts from end-users.

CustomAction Table Entry
Basic Custom Action Type 6 decimal (0x06 hex)
Source Column Key to the Binary table identifying the binary data stream which represents the VBScript file.
Target Column The name of an optional script function.

VBScript File Installed With a Product

In cases where the installed product may use the same VBScript file, it may make sense to avoid duplication, and run the VBScript custom action using the installed file. This does restrict the points in the installation process where the custom action may be used. Mainly this includes any point after the InstallFiles standard action during a normal installation, or prior to the RemoveFiles standard action during an uninstallation.

CustomAction Table Entry
Basic Custom Action Type 22 decimal (0x16 hex)
Source Column Key to the File table identifying the VBScript file that is installed as part of the product.
Target Column The name of an optional script function.

VBScript Text Stored in This Sequence Table

For cases where the VBScript custom action consists of a relatively small amount of script text, it may be easier to store the script directly in the definition of the custom action. This basic custom action type facilitates this feature, storing the VBScript text in the Target field of the custom action. Due to field size restrictions, the VBScript text is limited to a maximum length of 255 characters.

CustomAction Table Entry
Basic Custom Action Type 38 decimal (0x26 hex)
Source Column null
Target Column Literal VBScript text

VBScript Text Specified by a Property Value

An alternative method for storing the VBScript text for a custom action is to place it in an entry in the Property table. That property can then be referenced by the custom action. Property values can accommodate longer text than the Target field of the CustomAction table, so longer scripts can be used in this case. However, for maintenance purposes, it is generally best to store relatively short scripts in the Property table, and to use the Binary table for longer scripts.

CustomAction Table Entry
Basic Custom Action Type 54 decimal (0x36 hex)
Source Column Key to the Property table identifying the property containing the script text.
Target Column The name of an optional script function.

Terminating an Installation

Under certain conditions, it may be necessary to abort an installation. Windows Installer provides a special custom action type for this purpose.

Display an Error Message

This basic custom action type allows the install author to present an error message to the user and terminate the installation. The error message displayed can be a string literal or retrieved from the Error table.

CustomAction Table Entry
Basic Custom Action Type 19 decimal (0x13 hex)
Source Column null
Target Column Formatted text string representing the error message to display. If the formatted text evaluates to an integer, that number is used as an index into the Error table to retrieve the message to display.

Setting Property Values

Properties are the variables for the Windows Installer, and there are many cases where it is necessary to define new properties based on certain installation attributes. This can be accomplished through custom actions that use the capabilities of the Formatted data type to set the values of properties and directories.

Directory Set With Formatted Text

Directories are really just a special kind of properties. A directory can be used virtually anywhere a property is expected, though the reverse is not necessarily true. In certain cases, it may be desired to change the value of a directory defined in the Directory table based on attributes of the current installation session. For example, a directory could be set to match the value read from an environment variable.

While any formatted text string can be entered, and will pass MSI validation, problems may be encountered at install-time if the formatted text does not resolve to a properly formatted directory path. Note that directory paths must always end with a directory separator (\).

CustomAction Table Entry
Basic Custom Action Type 35 decimal (0x23 hex)
Source Column Key to the Directory table identifying the directory that will be set using the formatted text.
Target Column A text string formatted according to the rules of the Formatted data type.

Property Set With Formatted Text

Unlike directories, properties have no formatting or length restrictions. As such, any formatted text can be used for setting the property value.

CustomAction Table Entry
Basic Custom Action Type 51 decimal (0x33 hex)
Source Column Key to the Property table identifying the property that will be set using the formatted text.
Target Column A text string formatted according to the rules of the Formatted data type.

Return Processing

Two bit flags in the custom action type are related to return processing. One flags controls whether the exit code returned from the custom action will be checked, while the other specifies whether the Windows Installer will wait for the custom action to complete before continuing with the installation process.

Ignore Exit Code

Bit Flag Value 64 decimal (0x40 hex)

If this flag is set, the Windows Installer will act as though the custom action returned an exit code indicating successful completion. This flag has no effect on basic custom action types that set directory/property values (types 35/51) or that terminate the installation (type 19). If this flag is not set, then DLL and EXE-based custom actions must return one of the following values:

Return Constant Name

Constant Value

Description
ERROR_FUNCTION_NOT_CALLED

1626

The custom action was not executed
ERROR_SUCCESS

0

The custom action completed successfully
ERROR_INSTALL_USEREXIT

1602

The user terminated the installation prematurely
ERROR_INSTALL_FAILURE

1603

An unrecoverable error occurred
ERROR_NO_MORE_ITEMS

259

Skips remaining actions, not an error

JScript and VBScript custom actions must return one of the following values if the Ignore Exit Code flag is not set:

Return Constant Name

Constant Value

Description
msiDoActionStatusNoAction

0

The custom action was not executed
msiDoActionStatusSuccess

IDOK=1

The custom action completed successfully
msiDoActionStatusUserExit

IDCANCEL=2

The user terminated the installation prematurely
msiDoActionStatusFailure

IDABORT=3

An unrecoverable error occurred
msiDoActionStatusSuspend

IDRETRY=4

Suspended sequence to be resumed later
msiDoActionStatusFinished

IDIGNORE=5

Skips remaining actions, not an error

Asynchronous Execution

Bit Flag Value 128 decimal (0x80 hex)

By default, the Windows Installer waits for a custom action to complete before proceeding with the installation. When the Asynchronous Execution flag is set, the Windows Installer will immediately continue with further installation actions after starting the custom action. This bit flag cannot be used on rollback custom actions or script custom actions (JScript/VBScript).

Additionally, this bit flag affects the behavior associated with the Ignore Exit Code bit flag. When Asynchronous Execution is enabled for a custom action, any checking of custom action exit codes will be delayed until the end of the installation sequence. If both Ignore Exit Code and Asynchronous Execution are set, then the custom action will be allowed to continue even after the Windows Installer terminates the installation session. Because of this, DLL custom actions cannot have both of these bit flags set. Only EXE-based custom actions are allowed to have both bit flags set.

Scheduling Options

When it comes to scheduling custom actions, there are two main options to be considered. Custom actions can either be set to run immediately, or they can be scheduled to run at a later point when the installation script is processed. In general, custom actions can be invoked from any of the following points in the installation process:

Immediate Custom Actions

Immediate custom actions are executed as soon as they are encountered in a sequence table, UI control event or explicit call to MsiDoAction. As a class, immediate custom actions can appear at any point in the installation process where custom actions are valid, though certain basic types of custom actions may have further restrictions on when they can be executed.

First Sequence

Bit Flag Value 256 decimal (0x100 hex)

By default, immediate custom actions will be executed each time they are encountered in a sequence table. By setting the First Sequence bit flag, the install author can direct the Windows Installer to only execute the custom action the first time it is encountered in a sequence table. Thus if a custom action appears in both the InstallUISequence and InstallExecuteSequence, it will be executed in the InstallUISequence when the install is run with full UI mode, or in the InstallExecuteSequence when the install is run with less than full UI mode.

Once Per Process

Bit Flag Value 512 decimal (0x200 hex)

The Once Per Process flag is similar to the First Sequence flag. In fact, on Windows 9x machines, both flags will result in the same behavior. The reason for this is that under Windows 9x, the InstallUISequence and InstallExecuteSequence tables are executed within the same process. With NT-based Windows (2000/XP/Vista/7), the InstallUISequence is executed in a client process while the InstallExecuteSequence is executed in a service process.

The primary purpose of this flag is to prevent actions that modify session state, such as property and database data, from running more than once. Thus if a custom action with this flag appears in both the InstallUISequence and InstallExecuteSequence, it will only be executed in the InstallUISequence on Windows 9x systems, but will be executed in both sequences on NT-based Windows systems.

Client Repeat

Bit Flag Value 768 decimal (0x300 hex)

This flag is actually the combination of the First Sequence and Once Per Process bit flags. Since it does not make sense to have both of those flags turned on at the same time, the Windows Installer creators decided to use that bit flag combination for a different meaning, rather that use a separate bit flag.

The Client Repeat bit flag setting will cause the custom action to execute only if it is running on the client after the UI sequence has been run. Thus, this is valid only for custom actions listed in the InstallExecuteSequence.

In-Script Custom Actions

Those custom actions scheduled in the InstallExecuteSequence between InstallInitialize and InstallFinalize may be flagged as In-Script custom actions (also known as deferred custom actions). These custom actions will not be processed immediately, but instead will be added to the installation script for later processing.

Deferred

Bit Flag Value 1024 decimal (0x400 hex)

This is the bit flag that identifies the custom action as In-Script or deferred, instead of immediate. Any custom actions with this bit set will be scheduled for later processing in the installation script. The exact point of that later scheduling may depend upon the settings of other bit flags.

Rollback

Bit Flag Value 256 decimal (0x100 hex)

The rollback bit flag must be used in combination with the deferred bit flag, and cannot be set in combination with the commit bit flag. It identifies the custom action as a rollback action. This causes the custom action to be scheduled for possible later processing in the rollback phase of the installation script.

Commit

Bit Flag Value 512 decimal (0x200 hex)

The commit bit flag must be used in combination with the deferred bit flag, and cannot be set in combination with the rollback bit flag. It identifies the custom action as a commit action. This causes the custom action to be scheduled for possible later processing in the commit phase of the installation script.

No Impersonate

Bit Flag Value 2048 decimal (0x800 hex)

The No Impersonate bit flag must be used in combination with the deferred bit flag. Normally, deferred custom actions are executed with user impersonation, such that they run with the privileges of the logged in user. In some cases, the user privileges are insufficient to perform certain system modifications or queries, so elevated privileges are needed. The No Impersonate bit flag can be used to cause the custom action to run with system privileges, thus providing a greater level of privileges.

TS Aware

Bit Flag Value 16384 decimal (0x4000 hex)

The TS Aware bit flag must be used in combination with the deferred bit flag. This bit flag has no effect if set in combination with the No Impersonate bit flag. Normally, deferred custom actions will execute with no user impersonation when running a per-machine installation on a terminal server. Setting the TS Aware bit flag will cause the custom action to run with user impersonation during a per-machine install on a server running the Terminal Server role service.

Custom Action Type Bit Map

The following table contains a summarization of the bit mapping used for Windows Installer custom action type values:

Bit

Usage

0-5

Basic Custom Action Type

6

Ignore Exit Code

7

Asynchronous Execution

8

First Sequence (immediate custom actions) / Rollback (deferred custom actions)

9

Once Per Process (immediate custom actions) / Commit (deferred custom actions)

10

Deferred

11

No Impersonate

12-13

unused

14

TS Aware

15-31

unused

Windows Installer Custom Action Type Encoder/Decoder

As an aid for encoding and decoding Windows Installer custom action types, I have created an Excel spreadsheet application for this purpose. This spreadsheet can be found here.

About Daniel Brannon

Daniel Brannon founded OSoSLO in 2009 to provide software tools and services for business and technology professionals.
This entry was posted in Windows Installer. Bookmark the permalink.

Leave a Reply