While recently working on moving an FTP server (using Passive FTP) from on-premises to Azure, I needed to expose the FTP server to the internet via an Azure External Load Balancer. A few things were not well documented. I thought it would be good to put together a step-by-step guide on setting up the load balancer, configuring the FTP server on the backend, and setting the Network Security Group rules associated with the FTP server. In this example, the FTP server will be running on IIS, and it is assumed that there is no Azure Firewall\NVA associated with the VNET that is hosting the FTP server. Suppose your environment does have an Azure Firewall\NVA. In that case, there will be an additional Azure resource needed, an Azure Route Table with a\0 out to the internet associated with the subnet the FTP server is hosted. This is required in order to route the outbound traffic back out to the internet. If this is not done the outbound FTP traffic will die on the Azure Firewall\NVA.

Windows IIS FTP Server Configuration

I won’t be going into the details of the FTP server configuration here. I will just be going over the FTP firewall settings and set the Data Channel Port (Passive FTP Ports). Below is how the FTP server is configured to support the external load balancer. On the IIS server settings – FTP Firewall Support is where you define the Data Channel Port Range.

FTP Firewall Support

I have defined ports 5000-5002.

Define Ports

At the site, the level is where you configure the public IP address of the external load balancer.

FTP Testt

As you see in the screenshot below the public IP of the external load balancer is configured.

FTP Firewall Support

Azure External Load Balancer

Configuring an Azure External Load Balancer for FTP is strait forward. One thing to note with Passive FTP is that all the Data Channel ports for FTP must be defined in the load balancing rules. If you define ports 5000-5100, there will be 100 load balancing rules for each data channel port. I recommend keeping the number of passive ports to a minimum since Azure External Load Balancers do not support a port range.


  • Azure Public IP – this will be configured as the load balancer’s front-end IP
  • Azure Load Balancer
  • FTP Port requirements for inbound traffic and public IP address(es) of the client(s) that will be accessing the FTP server

Deploy the Load Balancer

Search for Azure Load Balancer in the search bar in the Azure Portal

Search for Azure Load Balancer
Select Create

Creating a Load Balancer in Azure

Define the following parameters then select Next: Frontend IP configuration

  • Resource Group
  • Name
  • Region
  • Leave SKU: Standard
  • Leave Type: Public
  • Tier: Region

Project Details Load Balancing

Select Add a frontend IP Configuration, define the following parameters

  • Frontend IP Name
  • IP version: leave IPv4
  • IP type: leave IP address
  • Public IP address
  • Gateway Load balancer: leave None

Select Add

Name Load Balancer ad Public IP Address

Select Review + create – we will go over the configuration of the remaining items of the Load Balancer after it has been deployed.

Add Backend Pool

Configuration of the Azure External Load Balancer

I will now go over the configuration of the Azure Load Balancer. This will detail how the Backend pool, Health probes, Load balancing rules, and Outbound rules are configured.

Configuration in the Azure External Load Balancer

Frontend IP Configuration

As you see in the screenshot below, the frontend IP is defined by the rules associated with the front end. After additional load balancing rules are added I will review the frontend configuration again.

Frontend IP configuration

Backend Pools

The backend pool configuration is how you associate a virtual machine with the load balancer. In the screenshot below, I use the following configuration

  • Virtual network – the VNET that the target virtual machine is associated with
  • Backend Pool Configuration: NIC
  • IP Version: IPv4
  • Add the virtual machine by selecting the +Add button
  • Select Save

Add virtual machines to backend pools

Add Virtual Network

Health Probes

I have created an FTP Health probe for port 21. I will also be using this health probe for my FTP data channel ports. You can make a health probe for each data channel port.

  • Name: FTP
  • Protocol: TCP
  • Port: 21
  • Interval: 5
  • Unhealthy threshold:2

Use Health Probe for FTP Data

Load Balancing Rules

I have the port 21 load balancing rule already configured but need to add the FTP data channel ports that I have defined in FTP Firewall (5000-5002).

Define the following parameters:

  • Name: I like to give it the name of the port number, in this example, I will use 5000
  • IP Version: IPv4
  • Frontend IP address: Select your Frontend IP Config – FEIP
  • Backend Pool: Select your backend pool – BEP-01
  • Protocol: TCP
  • Port: 5000
  • Backend Port: 5000
  • Health Probe: ftp (TCP:21)
  • Session persistence: None
  • Idle timeout (minutes): 4
  • TCP reset: Disabled
  • Floating IP: Disabled
  • Outbound source network address translation (SNAT): (Recommended) Use outbound rules to provide backend pool members access to the internet
  • Select Add

Here is the full set of Load Balancing rules:

Load Balancing Rules

How to Add a Loan Balancing Rule

Outbound Rules

There is a requirement to create an outbound rule for the back-end resource to be able to reach out to the internet.

Create an outbound rule for back-end resource

Network Security Group Rules

The final step in configuring this solution is configuring the inbound rules for the NSG. The rule should have TCP ports 21 and the data channel ports defined in your FTP firewall (in my case, this would be 5000-5002).

External IP client accessing the FTP server


Passive FTP via an Azure Load Balancer combined with Network Security Group is a great solution to expose your FTP server on the internet securely. Understanding the total possible number of connections at a single time will help determine the minimum number of passive data channel ports required to configure on the Azure External Load Balancer.

A DoD client requested support with automated file transfers. The client has files placed in a common folder that can be accessed by the standard File Transfer Protocol (FTP). Given the FTP server’s connection information, the client requested the files to be moved to an Amazon Web Services (AWS) S3 bucket where their analysis tools are configured to use.

Automating the download and upload process would save users time by allowing for a scheduled process to transfer data files. This can be achieved using a combination of AWS Lambda and EC2 services. AWS Lambda provides a plethora of triggering and scheduling options and the power to create EC2 instances. By creating an EC2 example, a program or script can avoid Lambdas’ limitations and perform programmatic tasking such as downloading and uploading. Additionally, this can be done using Terraform to allow for deployment in any AWS space.

Writing a Script to Do the Work

Create a Script that can log in to the FTP server, fetch/download files, and copy them to an S3 bucket before using Terraform or AWS console. This can be done effectively with Python’s built-in FTPlib and the AWS boto3 API library. There are various libraries and examples online to show how to set up a Python script to download files from an FTP server and use the boto3 library to copy them to S3.

Consider writing the script that file size will play a significant role in how FTPlib and Boto3’s copy functions work. Anything over 5GB will need to be chunked from the FTP Server and use the multiple file upload methods for the AWS API.

Creating an Instance with Our Script Loaded

Amazon provides Amazon Managed Images (AMI) to start up a basic instance. The provided Linux x86 AMI is the perfect starting place for creating a custom instance and eventually custom AMI.

With Terraform, creating an instance is like creating any other module, requiring Identity and Access Management (IAM) permissions, security group settings, and other configuration settings. The following shows the necessary items needed to make an EC2 instance with a key-pair, permissions to write to s3, install Python3.8 and libraries, and copy the script to do the file transferring into the ec2-user directory.

First, generating a key-pair, a private key, and a public key is used to prove identity when connecting to an instance. The benefit of creating the key-pair in the AWS Console is access to the generated .pem file. Having a local copy will allow for connecting to the instance via the command line, while great for debugging, but not great for deployment. Terraform can be generated and store a key-pair in its memory to avoid passing sensitive information.

# Generate a ssh key that lives in terraform
# https://registry.terraform.io/providers/hashicorp/tls/latest/docs/resources/private_key
resource "tls_private_key" "instance_private_key" {
  algorithm = "RSA"
  rsa_bits  = 4096

resource "aws_key_pair" "instance_key_pair" {
  key_name   = "${var.key_name}"
  public_key = "${tls_private_key.instance_private_key.public_key_openssh}"


To set up the secrSetup, which is the security group to run the instance in, open up the ports for Secure Shell (SSH) and Secure Copy Protocol (SCP) to copy the script file(s) to the instance. A security group acts as a virtual firewall for your EC2 instances to control incoming and outgoing traffic. Then, open other ports for ingress and egress as needed, i.e. 443 for HTTP traffic. The security group will require the vpc_id for your project. This is the Visual Private Cloud (VPC) that the instance will be running. The security group should match up with your VPC settings.

resource "aws_security_group" "instance_sg" {
  name   = "allow-all-sg"
  vpc_id = "${var.vpc_id}"
  ingress {
    description = "ftp port"
    cidr_blocks = [""]
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"

The IAM policy, for instance, will require PutObject access to the S3 bucket. The Terraform module will need the S3 bucket as an environment variable, and a profile instance is created. If creating the IAM policy in the AWS Console, a profile instance is automatically created, but it has to be explicitly defined in Terraform.

#iam instance profile setup
resource "aws_iam_role" "instance_s3_access_iam_role" {
  name               = "instance_s3_access_iam_role"
  assume_role_policy = <<EOF
  "Version": "2012-10-17",
  "Statement": [
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      "Effect": "Allow",
      "Sid": ""
resource "aws_iam_policy" "iam_policy_for_ftp_to_s3_instance" {
  name = "ftp_to_s3_access_policy"

  policy = <<EOF
  "Version": "2012-10-17",
  "Statement": [
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
      "Resource": "arn:aws:s3:::${var.s3_bucket}"

resource "aws_iam_role_policy_attachment" "ftp_to_s3" {
  role       = aws_iam_role.instance_s3_access_iam_role.name
  policy_arn = aws_iam_policy.iam_policy_for_ftp_to_s3_instance.arn

resource "aws_iam_instance_profile" "ftp_to_s3_instance_profile" {
  name = "ftp_to_s3_instance_profile"
  role = "instance_s3_access_iam_role"

Defining the instance to start from and create the custom AMI from in the Terraform will need the following variables:

  • AMI – the AMI of the Linux x86 image
  • instance_type – the type of instance, i.e., t2.micro
  • subnet_id – the subnet string from which VPC the instance will run on
  • key-name – the name of the key, should match the key-pair name generated above or the one from the AWS console, could use a variable reference here too

Define the connection and provisioner attributes to copy the python script to do the file transferring to the ec2-user home folder. The connection will use the default ec2-user using the secure key and then copy over the python file. If using the key downloaded from AWS Console, use the following to point to the file private_key = “${file (“path/to/key-pair-file.pem”)}”.

Complete the instance setup with the correct Python version and library. The user_data attribute sends a bash script to install whatever is needed— in this case, updating Python to 3.8, installing the boto3, and paramiko libraries.

# Instance that we want to build out
resource "aws_instance" "ftp-to-s3-instance" {
  ami           = var.ami
  instance_type = var.instance_type
  subnet_id     = var.subnet_id
  key_name 	   = "${var.key_name}" #use your own key for testing
  security_groups      = ["${aws_security_group.instance_sg.id}"]
  iam_instance_profile = "${aws_iam_instance_profile.ftp_to_s3_instance_profile.id}"

  # Copies the python file to /home/ec2-user
  # depending on how the install of python works we may need to change this location
  connection {
    type        = "ssh"
    user        = "ec2-user"
    host        = "${element(aws_instance.ftp-to-s3-instance.*.public_ip, 0)}"
    private_key = "${tls_private_key.instance_private_key.private_key_pem}"

  provisioner "file" {
    source      = "${path.module}/ftp_to_s3.py"
    destination = "/home/ec2-user/ftp_to_s3.py"

  user_data = <<EOF
sudo amazon-linux-extras install python3.8
python3.8 -m pip install -U pip
pip3.8 --version
pip3.8 install boto3 
pip3.8 install paramiko 


The last step is to create the custom AMI. This will allow our Lambda to duplicate and make as many of these instances as need.

resource "aws_ami_from_instance" "ftp-to-s3-ami" {
  name               = "ftp-to-s3_ami"
  description        = "ftp transfer to s3 bucket python 3.8 script"
  source_instance_id = "${aws_instance.ftp-to-s3-instance.id}"

  depends_on = [aws_instance.ftp-to-s3-instance]

  tags = {
    Name = "ftp-to-s3-ami"

Creating Instances on the Fly in Lambda

Using a Lambda function that can be triggered in various ways is a straightforward way to invoke EC2 instances. The following python code show passing in environment variables to be used in an EC2 instance as both environment variables in the instance and arguments passed to the Python script. The variables needed in the python script for this example are as followed:

  • FTP_HOST – the URL of the FTP server
  • FTP_PATH – the path to the files on the URL server
  • FTP_USERNAME, FTP_PASSWORD, FTP_AUTH – to be used for any authentication for the FTP SERVER
  • S3_BUCKET_NAME – the name of the bucket for the files
  • S3_PATH – the folder or path files should be downloaded to in the S3 bucket
  • Files_to_download – for this purpose, a python list of dictionary objects with filename and size to downloaded.

For this example, the logic for checking for duplicate files is down before the Lambda invoking the instance for transferring is called. This allows the script in the instance to remain singularly focused on downloading and uploading. It is important to note that the files_to_download variable is converted to a string, and the quotes are made into double-quotes. Not doing this will make the single quotes disappear when passing to the EC2 instance.

The init_script variable will use the passed-in event variables to set up the environment variables and python script arguments. Just like when creating the instance, the user_data script is run by the instance’s root user. The root user will need to use the ec2-user’s python to run our script with the following bash command: PYTHONUSERBASE=/home/ec2-user/.local python3.8 /home/ec2-user/ftp_to_s3.py {s3_path} {files_to_download}.

# convert to string with double quotes so it knows its a string
    files_to_download = ",".join(map('"{0}"'.format, files_to_download))
    vars = {
        "FTP_HOST": event["ftp_url"],
        "FTP_PATH": event["ftp_path"],
        "FTP_USERNAME": event["username"],
        "FTP_PASSWORD": event["password"],
        "FTP_AUTH_KEY": event["auth_key"],
        "S3_BUCKET_NAME": event["s3_bucket"],
        "files_to_download": files_to_download,
        "S3_PATH": event["s3_path"],

    init_script = """#!/bin/bash
                /bin/echo "**************************"
                /bin/echo "* Running FTP to S3.     *"
                /bin/echo "**************************"
                export S3_BUCKET_NAME={S3_BUCKET_NAME}
                export PRODUCTS_TABLE={PRODUCTS_TABLE}
                export FTP_HOST={FTP_HOST}
                export FTP_USERNAME={FTP_USERNAME}
                export FTP_PASSWORD={FTP_PASSWORD}
                PYTHONUSERBASE=/home/ec2-user/.local python3.8 /home/ec2-user/ftp_to_s3.py {s3_path} {files_to_download}
                shutdown now -h""".format(

Invoke the instance with the boto3 library providing the parameters for the custom image AMI, Instance type, key-pair, subnet, and instance profile, all defined by Terraform environment variables. Optionally, set the Volume size to 50GB from the default 8GB for larger files.

instance = ec2.run_instances(
        IamInstanceProfile={"Arn": INSTANCE_PROFILE},
        BlockDeviceMappings=[{"DeviceName": "/dev/xvda", "Ebs": {"VolumeSize": 50}}],


After deploying to AWS, Terraform will have created a Lambda that invokes an EC2 instance running the script passed to it during its creation. Triggering the Lambda function to invoke the custom instance can be done from a DynamoDB Stream update, scheduled timer, or even another Lambda function. This provides flexibility on how and when the instance is called.

Ultimately, this solution provides a flexible means of downloading files from an FTP server. Changes to the Lambda invoking the instance could include separating the file list to create several more minor instances to run simultaneously, moving more files faster to the AWS S3 bucket. This greatly depends on the client’s needs and the cost of operating the AWS services.

Changes can also be made to the script downloading the files. One option would be to use more robust FTP libraries than the built-in provided python library. Larger files may require more effort as FTP servers can timeout when network latency and file sizes come into play. Python’s FTPlib does not auto-reconnect, nor does it keep track of incomplete file downloads.