McAfee ePolicy Orchestrator

Maintenance

Basic SQL maintenance can provide a measure of confidence in consistent SQL performance, a critical component of McAfee ePO performance. Because the SQL database for McAfee ePO is highly transactional in nature, the execution speed of those transactions directly relates to how fast the product is able to operate.

The Recommended Maintenance Plan (login required) for the McAfee ePO database focuses primarily on several configuration options and a regularly scheduled maintenance plan that consists of three tasks.

Configuration options recommended for the McAfee ePO SQL database:

• Simple recovery model. The full recovery model introduces additional overhead (backing up the transaction log) and requires a great deal more space on the SQL server for the ever-growing transaction log.
• Avoiding the “Shrink Database” option when possible. Shrinking the database causes database fragmentation, which reduces performance. The option to shrink the data file can be taken in situations such as when an uncharacteristically large flood of unexpected events is generated and stored within the SQL database. A large number of events might occur after a malware outbreak.

To configure a regularly scheduled SQL maintenance plan:

• Verify database integrity.
• Rebuild and reorganize indexes.
• Back up the SQL database. 

NOTE: Creating a maintenance plan is possible in full versions of Microsoft SQL. Microsoft SQL Express is a freely available, but limited, version of Microsoft SQL. Microsoft SQL Express is only recommended for use in environments with less than 1,500 managed systems.

Rebuilding and reorganizing indexes improves SQL performance. When the database is well-maintained, database size alone does not negatively affect query performance.

NOTE: KB67184, the Recommended Maintenance Plan, is a registered KB and requires access to the McAfee ServicePortal. In addition to what is outlined here, the Maintenance Plan document includes an SQL script optimized for rebuilding and reorganizing indexes in production environments.

Recommended Specifications

When determining the hardware specifications required for acceptable SQL performance in a McAfee ePO environment, consider the following:

• Size of environment: the current number of nodes the administrator plans to manage, accounting for potential or planned growth.
• Number and complexity of the McAfee products being managed in the environment.

NOTE: The SQL database can be hosted on the same machine that is responsible for hosting McAfee ePO, but this configuration is only recommended in environments with less than 10,000 managed nodes. In environments greater than 25,000 managed nodes, the SQL server should not only be separate, but also on a physical (as opposed to virtual) machine.

SQL performance itself is most closely tied to three specific hardware factors:

• Processing power: This is most commonly defined as the number of CPU cores.
• Disk I/O: How fast data can be written and read from the disks where SQL data and log files are stored.
• System memory: Memory itself does not speed up the database, but an SQL server that is bottlenecked on system memory will reduce performance as data is written to the hard drive via the Windows page file.

Hardware recommendations are difficult to estimate based on the factors described above, but in general, the SQL server should have similar, if not greater, hardware granted to it than the McAfee ePO server itself—especially memory in large environments. For example, in an environment with 75,000 to 150,000 endpoints, the standalone SQL server should have between 32GB and 128GB of memory installed, while the McAfee ePO server should have between 32GB and 64GB.

SQL Blocking

Blocking within the SQL database occurs when two transactions require access to the same resource—essentially, the transactions form a chain. When the head blocker is complete, the resource is made available again and the next transaction can move ahead.

Blocking, like SQL deadlocks, is a normal occurrence in a busy database. But, there are extreme scenarios that can arise when a blocking transaction runs for a long time. In that case, especially if the locked resource is commonly used (for example, the tables responsible for storing McAfee ePO System Tree data), the number of transactions waiting on the head blocker can grow. This condition can cause the McAfee ePO console to be unresponsive.

NOTE: Blocking issues are most frequently caused by inefficiencies in the design of objects within the SQL database—for example, a large table that has no index. Upgrades to McAfee ePO itself, or the point-product extension responsible for the database object, are often the first step.
Some more common blocking scenarios include:

• Active Directory Sync tasks configured to move or delete many systems.
• A backlog of client or threat events waiting to be parsed to the SQL database.

NOTE: In general, index fragmentation can make a significant difference in the amount of time it takes for a transaction and how long it locks a specific resource. Refer to the section on “SQL Maintenance” or KB67184 (login required) for instructions on how to ensure fragmentation is reduced.

Deadlocks

An SQL deadlock is a condition where two transactions are both blocked, waiting for the other transaction to complete its work and release a lock. Contention for the shared resource leads to one of the transactions being chosen as the “deadlock victim.” The transaction is killed and the deadlock victim may be immediately retried depending on the action.

NOTE: Deadlocks, while preferably avoided entirely, are in general a natural occurrence in a busy SQL database. Deadlocks must be addressed when they are consistent or impede McAfee ePO functionality.

Deadlocks are not clearly displayed from within the McAfee ePO console. Instead, the console might display an unexpected error or other similar problem. Deadlocks are logged in the Application Server service’s primary log, the orion.log, located in the ePO_install_dir\server\logs folder.

When deadlocks have been confirmed as a problem, it’s necessary to collect more data to resolve the problem. KB90786 (login required) describes the three methods of enabling and capturing deadlock trace information, which is necessary to provide to McAfee Support when troubleshooting a persistent deadlock problem.

Database Size & Disk Space

SQL database size considerations fall under three large branches:

• SQL data file size limitations. This is either a configuration issue (Auto Growth disabled) or a limitation of SQL Express, the free version of Microsoft SQL (10GB file size limit in versions since 2008R2).

NOTE: Database file size does not itself lead to performance problems, even in very large environments, if the SQL database is properly maintained and defragmented. SQL Express instances are commonly responsible for administrators experiencing disk size problems. With SQL Express it is necessary to first identify what is responsible for taking up the most space in the database and then remove the responsible events manually or, preferably, via McAfee ePO-configured Server Tasks.

• Tempdb growth. Tempdb is a shared system database (all databases installed and mounted to an SQL instance use the same tempdb) and grows on a need basis. For quick remediation of an enormous tempdb, restart the Microsoft SQL service—tempdb is rebuilt on service start. For more detailed information about what tempdb does and how to troubleshoot unexpected growth within tempdb, see KB90101. 

NOTE: McAfee ePO 5.3.0, 5.3.1, and 5.3.2 are most commonly impacted by tempdb growth problems because they use an SQL configuration option called Snapshot Isolation. Upgrading to McAfee ePO 5.3.3 or 5.9.1 is recommended.

• Transaction log file size limitations. Using the Simple recovery model (login required) should prevent this issue, but truncating the transaction log is also possible.

Disk space is also a concern during a McAfee ePO upgrade—the database can double in size during the installation process because of schema changes (especially when upgrading from McAfee ePO 5.3.x to 5.9.x and 5.10.x). KB79561 covers important considerations for preparing the database to reduce the impact of the database growth before an upgrade.

NOTE: Hosting the SQL database files on Windows mount points is increasing in popularity among SQL administrators. The default disk space checks that the McAfee ePO installer leverages cannot accurately detect available space on mount points; for this reason it’s necessary to use an installation switch to bypass the check.

Agent-Server Communication

The McAfee Agent is the client-side software that facilitates management of endpoints. Agent-server communication is the method by which system properties are uploaded to the server and new policies and client tasks are requested. When agent-server communication interval (ASCI) fails, the system is essentially unmanaged.

NOTE: McAfee Agent 5.x logs important ASCI information in masvc_systemname.log. For detailed information on Agent log locations including non-Windows platforms, see KB82170.

If the ASCI fails between the client (MA) and the server (McAfee ePO), review the logs from each. Use KB90603 for instructions on how to read the logs.
Commonly encountered problems with ASCI:

• Third-party .DLL injection. See KB88085.
• The McAfee ePO server/Agent Handler reaching a maximum connections state. See KB77680.
• Corruption during the Agent property collection process. See KB88041.

How to troubleshoot agent-server communication failures in McAfee Agent 5.x includes many more common causes and solutions for agent-server communication failures.

Console Performance

When navigating or taking action in the McAfee ePO console is slow, there are several simple but critical checks to verify:

• First, confirm that the McAfee ePO server meets or exceeds the minimum requirements detailed in the Installation Guide. For example, see the McAfee ePO 5.10 Installation Guide.

NOTE: The minimum requirements detailed in documentation are just that—minimum. Depending on environment size and complexity or number of products installed and managed, your McAfee ePO or SQL server might need to be significantly more powerful than the minimum requirements.

• Verify that the McAfee ePO server’s SQL database is not heavily fragmented. SQL database fragmentation is a top cause of console performance issues because nearly all console actions transact within the database. The McAfee recommended SQL maintenance plan and a reindexing (maintenance) SQL script, is detailed in KB67184 (login required).
• Check the McAfee ePO console’s Server Task Log and determine if there are many “In Progress” Server Tasks. If so, the server might be experiencing an issue with the internal dbclean task, which is responsible for terminating tasks in a timely manner. For more information on this issue, see the section titled “Server Tasks Running Indefinitely” in KB67184 (login required).

Beyond these initial checks, troubleshooting a McAfee ePO console performance issue can be a daunting task, and will likely involve contacting McAfee Technical Support. If the administrator is comfortable working with SQL queries, see KB90786 (login required) for advanced troubleshooting and data collection techniques, including identifying the cause of SQL blocking and enabling the configuration necessary to track SQL deadlock issues.

Repository Replication

Distributed repositories are used to provide additional sources for clients to receive both content updates (.DATs) and product deployments. When repository replication fails on a consistent basis, the impacted repositories do not match the McAfee ePO Master Repository. As a failsafe mechanism, update attempts from said repository are blocked and fail.

McAfee ePO contains a default Automatic Response that can be configured to alert the administrator via email if a replication fails: “Distributed Repository Replication fails.” If the administrator becomes aware of repeated failures to replicate to a specific repository, action must be taken to remediate.
Initial steps to begin troubleshooting a replication failure:

• Identify the name and type of the repository (UNC, SuperAgent, etc.).
• Review Server Task log entries demonstrating the failure.
• Test a manual, on-demand replication to determine if the problem is consistent.

McAfee ePO logs detailed information about repository replication, including critical failure messages, in the ePOAPSvr_servername.log, located in the ePO_install_dir\DB\Logs directory. More complete information regarding ePO server logs is located in the McAfee ePO 5.10 Log File Reference Guide.
Common causes of repository replication failures include:

• Invalid credentials (password changed, account locked, etc.) saved in the McAfee ePO configuration page for a UNC repository. Check the Distributed Repository page and edit the configuration for the problematic repository—a “test connection” button is available for verification.
• HTTP 403 errors when replicating to a SuperAgent repository (SADR) is a known issue with McAfee Agent 5.x versions earlier than 5.0.5.
• Several different defects exist that can cause the McAfee ePO Application Server service (Tomcat) to crash during a repository replication. For more information, see KB90753 and KB89590.

Server Tasks Running Indefinitely

McAfee ePO includes an internal, hidden server task called the dbclean task, which is responsible for terminating tasks that have reached their expiration time. There are several common issues with the dbclean task that can lead to the same symptom, such as many Server Tasks running indefinitely or far past their standard expiration time.

NOTE: The one exception to the above is Run Client Task Now. These tasks function differently than other server tasks—namely, they are dependent on receiving a DataChannel message from the endpoint to report on status. If this message fails to be sent or received, the tasks can stay in progress until the McAfee ePO Application Server service is restarted. For details on what this looks like and why it happens, see KB85013.

• KB83384 covers a scenario where the dbclean task fails to run as expected.

NOTE: While KB83384 lists McAfee ePO 5.1.2 and 5.3.1 as having resolved this problem, that refers to only one potential cause of the dbclean task becoming inactive. If the problems an administrator is experiencing match the symptoms described in the article, it’s still worth checking the status of the dbclean task via the SQL queries listed.

• KB84114 details a problem where the dbclean task itself is entirely missing from the SQL database.

Software Manager

The Software Manager (rebranded as Software Catalog in McAfee ePO 5.10 and higher) facilitates product downloads and updates. It requires an unobstructed connection to three McAfee hosted sites over port 443 to download the installers, updates, and extensions listed:

• lc.mcafee.com
• s-download.mcafee.com
• epo.mcafee.com

For a full list of ports required by McAfee ePO and its core components, see KB66797.

NOTE: Not every release is listed in the Software Manager. Often new releases are posted only to the Product Downloads site until published at a later date to the Software Manager. Contact support if packages are expected to be listed but do not appear or downloads fail, but not if the package has simply never been posted to the Software Manager.

General tips for Software Manager navigation and usage are described in KB71608, including descriptions of easily misunderstood UI behavior. Most of these quirks have been addressed in McAfee ePO 5.10.

If the Software Manager is blank or displays an error such as “The Product Catalog has not been downloaded yet,” the certificate used to securely connect to the McAfee backend server may have failed to download. For a link to download the certificate and steps to import it manually, see KB74029.

Duplicate Systems

When a single system is displayed multiple times in the McAfee ePO System Tree, it is considered a duplicate system. There are several reasons for duplicate systems—some legitimate, some not.

The first step when looking into the root cause of a duplicate systems issue is to determine if the duplicate entries are managed or unmanaged. Unmanaged entries have no system details—for example, no GUID, IP address, etc.

NOTE: Unmanaged duplicates are most commonly caused by an AD Sync, especially in McAfee ePO 5.3.3 and 5.9.0, which have a known issue that is resolved in McAfee ePO 5.9.1.

Managed duplicates most often arise from a situation where McAfee ePO receives properties from a managed endpoint and is unable to associate it with an existing entry in the System Tree. In other words, McAfee ePO believes the system to be new. In the standard (non-VDI mode) configuration, McAfee ePO uses the Agent GUID and MAC Address to identify existing machines, so a /forceinstall of the Agent can lead to a new GUID being generated, and potentially a new duplicate system entry.  

NOTE: If the administrator can “checkbox” one of the managed duplicate System Tree entries and more than one of the duplicate entries are selected, this indicates that there isn’t a duplicate system at all. Instead, McAfee ePO is forced to spread a single system’s properties across multiple rows because of an unexpectedly duplicated property. For example, if McAfee VirusScan Enterprise 8.8 Patch 8 and 9 are both installed and reporting at the same time. To isolate what property is at fault, reset the column view to default (Actions – Choose Columns – Use Default).

Repository Pull

The Master Repository Pull server task is a critical default action that is responsible for downloading content updates (.DATs, etc.) from McAfee-hosted source sites (McAfee HTTP) and checking it into the Master Repository on a regular interval.

Like distributed repository replication, rare failure is relatively normal—for example, if the backend source site is being replicated to during the time of the administrator’s repository pull, the task can fail. It is considered a best practice to schedule the Master Repository pull to run multiple times throughout the day to lessen the impact of a single failure. If the next pull succeeds, .DAT deployment in the environment is delayed only an hour, for example, instead of an entire day.

Master Repository facts:

• Physically located in the McAfee ePO installation directory in the DB\Software\ folder.
• Contains three branches: Current, Previous, and Evaluation. These are only used for organization purposes; for example, software present in the Evaluation branch is not unlicensed.
• Content in the Master Repository is automatically saved and cached on remote Agent Handlers in the environment on a need basis, similar to the LazyCaching functionality of SuperAgent repositories.

In the rare event of a failed repository pull leading to repository corruption (unable to view packages listed in the Master Repository page, for example), consider rebuilding the Master Repository.

NOTE: Rebuilding the repository removes all content files and installer packages. Policies and client tasks are not impacted because this data is stored in the McAfee ePO database and tied to extensions, not the repository. It will be necessary, though, to replicate to all distributed repositories after completing a repository rebuild.

If the administrator has chosen to install McAfee ePO in an air-gapped environment, or one without access to our public repository on the web, it may be necessary to configure McAfee ePO to update from another internal server.

Another commonly requested configuration is allowing client machines to update from any McAfee ePO server in the environment. To accomplish this, it’s necessary to share the master repository keys across any McAfee ePO servers installed in the environment.

Duplicate MAC Addresses

McAfee ePO uses the MAC address of a client machine as a secondary matching mechanism—in other words, if the Agent GUID has changed, the McAfee ePO server can check existing System Tree entries for the client’s MAC address and associate the incoming properties with that entry. If both the GUID and MAC Address are not found in the McAfee ePO database, a brand-new System Tree entry is created.

If machines are legitimately reporting the same MAC address (for example, when communicating over a VPN concentrator or when deployed from a Hypervisor configured to assign the same MAC), it is undesirable for McAfee ePO to use the MAC address for matching purposes. This can be disabled entirely (though it is not recommended) or done so on an OUI-basis, as described in KB52949.

NOTE: Excluding a MAC OUI from being used for matching purposes does not stop a MAC address from being displayed multiple times (duplicated) as a property in the System Tree. The McAfee Agent collects information about the installed MAC address and reports it to McAfee ePO; this is expected in an environment where multiple machines have the same MAC address.

Duplicated MAC addresses can lead to issues that are often the exact opposite of a duplicate systems issue. Systems can appear to “disappear” from the System Tree, despite the Audit Log not recording any mention of the machine being deleted.

Starting with McAfee ePO 5.10, it is possible to add necessary MAC exclusions through the console directly, whereas previously it required direct SQL access. Leverage the new Add Virtual MAC Vendor page in McAfee ePO Server Settings.

Console Login or Page Not Displayed

The McAfee ePO console, hosted by the McAfee ePO Application Server service (Tomcat), is the portal through which all management features are possible. If access to the console is impossible, so is management of the environment.

NOTE: If the console is down, unless the other components of McAfee ePO (for example, the McAfee ePO Server service) are also nonfunctional, client machines will continue to communicate, send events and properties, and update their content if available.

Common and easily resolvable issues with console access are typically related to the browser and include:

• Browser caching problems, especially after a McAfee ePO upgrade. Try clearing the browser’s cache or accessing the console from a different machine.
• Internet Explorer security settings. Especially on server operating system (such as the McAfee ePO server itself), it might be necessary to add the McAfee ePO URL to the list of Trusted Sites in Internet Explorer, or even to temporarily disable Internet Explorer Enhanced Security Configuration.
• The custom console certificate has expired. For steps on how to revert to a certificate backup if the console certificate had previously been replaced, see KB72477. The default console certificate will not expire because the expiration date should be set to 30 years after its generation.

Troubleshooting more complex console issues often comes down to one of two scenarios:

• Partially installed extensions (also referred to as dependency errors).
• A database connectivity problem.

Both scenarios are covered in KB81737, though for more detailed information on troubleshooting a database connection issue, refer to the Database Connection Issues section.

If the administrator completes a Disaster Recovery process and is greeted by dependency errors, see KB81754.

Database Connection Issues

A persistent and responsive connection to the SQL database is critical for all McAfee ePO server and Agent Handler(s) functionality.

NOTE: McAfee ePO Best Practices define the required connection speed between a handler (including McAfee ePO) and the database to be less than 10 ms, round trip. If the required speed isn’t possible, the database or handlers must be moved to accommodate; otherwise, severely negative performance impacts will occur.

Most database connection problems can be resolved by the McAfee ePO administrator with a basic level of SQL knowledge. Common issues include:

• Database authentication account credentials have expired or are invalid. This is especially common when an AD user account has been configured at installation for database connectivity. Best practices detail using an SQL service account to avoid this issue entirely. SQL authentication is also beneficial in an environment where remote Agent Handlers will be installed in a DMZ or outside of the domain, as they will be unable to authenticate using domain credentials.
• The McAfee ePO database is hosted on an SQL Express instance, which has a 10GB database limitation. When that limit is reached, McAfee ePO is unable to function. If the database is confirmed to be at the 10GB limit, the administrator should first confirm why the database has hit this limit using KB76720. If threat and client events are taking up a majority of database space, they can be purged directly from SQL using KB68961.
• The SQL port has changed either through configuration or because it was previously set to be dynamic. Instructions on how to verify what port SQL is configured to use are included in KB66166.
• Network changes have occurred that are blocking communication between McAfee ePO and SQL servers. Test connectivity via test.udl to verify.

NOTE: The most effective way to verify basic SQL connectivity is to use a test.udl file. The benefit of test.udl is that it exists outside of the framework of McAfee ePO and can be used to demonstrate if McAfee ePO is experiencing a software problem, or if the database connection issue exists at the SQL (or network) level. policies and client tasks.