Scaling & performance with AWS

Below are some estimates of how long a scan takes for various file types.

If a large number of scanning requests hit File Storage Security all at once, multiple instances of the ScannerLambda will be invoked to process the requests in parallel.

The exact scan time depends on the AWS concurrency settings of the Lambda functions, and well as how many other Lambdas you have running at the same time in your AWS account.

The Lambda concurrency setting number includes not only the Lambdas that File Storage Security uses, but also all the other Lambdas used in the same AWS account. You can leave this setting at its default (set by AWS) and make sure the Lambda concurrency in this AWS account is sufficient for the File Storage Security scan.

Below are some estimates of how long a scan takes and how much concurrency was used for various numbers of zipped 10 MB files. The data in the table was collected in a deployment where there was one storage stack and one scanner in the same region.

When the shared or reserved concurrency is lower than the requirement for the used concurrency, delays can be expected due to throttles.

In addition, there are three Lambda concurrencies invoked in the scan process, namely, the BucketListener, Scanner, and PostScanActionTag Lambda concurrencies. When a burst of traffic occurs in an architecture with one S3 bucket and one scanner, the BucketListener Lambda needs more concurrency.

The AWS Lambda service has a default setting of 1000 total allowable concurrent executions, and File Storage Security's ScannerLambda function follows this configuration. Thus, if two File Storage Security scanners are deployed under the same AWS account in the same region, the 1000 concurrent executions will be shared among the scanners (they won't each get 1000). Additionally, the concurrent executions will be further split with any other Lambdas deployed under the same AWS account in the same region.

The ScannerLambda receives scan messages by setting up an event source mapping to the SQS ScannerQueue. The maximum number of batches that an event source mapping can process simultaneously is 1000, and the batch size for the ScannerLambda is 1. In other words, ScannerLambda can poll at most 1000 scan messages from the SQS ScannerQueue simultaneously, even if there are still allowable concurrent executions. The polling latency determines the actual amount of scan messages that can be handled simultaneously. If the latency is 1 second, the maximum number of scan messages that can be handled in 1 second is bound to 1000. When the file uploading speed is higher than that amount, the scan messages are queued in the SQS ScannerQueue.

File identification information is also scanned by the Trend Micro Global Smart Protection Server in the cloud. (See Architecture and flow for details.) This server is set up to handle very large loads from across the globe and will never cause a performance bottleneck.