{"id":4952,"date":"2022-09-30T10:56:43","date_gmt":"2022-09-30T08:56:43","guid":{"rendered":"https:\/\/blog.besharp.it\/?p=4952"},"modified":"2022-09-30T10:59:50","modified_gmt":"2022-09-30T08:59:50","slug":"gateway-load-balancers-build-your-custom-network-appliance-on-aws","status":"publish","type":"post","link":"https:\/\/blog.besharp.it\/gateway-load-balancers-build-your-custom-network-appliance-on-aws\/","title":{"rendered":"Gateway Load Balancers: build your custom network appliance on AWS"},"content":{"rendered":"\n
“If you gaze into the abyss, the abyss gazes also into you” (F. Nietzsche).<\/em><\/p>\n\n\n\n In our previous article about ELB tricks and tips<\/a>, with a particular focus on Application Load Balancers <\/strong>and Network Load Balancers<\/strong>. Today, we are focusing specifically on Gateway Load Balancers to see how they can help to observe and filter outbound network traffic using appliances.<\/p>\n\n\n\n In this scenario, we assume that we already have designed and implemented a centralized networking solution using a Transit Gateway, as described here<\/a>.<\/p>\n\n\n\n If you want to add your customized IDS or network filtering solution, you can configure routing tables<\/strong> to forward traffic to an ENI interface of an EC2 instance, but this solution is not highly available<\/strong> and scalable.<\/p>\n\n\n\n Gateway Load Balancers satisfy this need, offering a way to route Layer 3 traffic transparently to inline highly available EC2 instances, regardless of the protocol\/port used. If you use other load balancers, you are bound to listeners on a specified protocol\/port; for example, you can’t forward ICMP traffic.<\/p>\n\n\n\n There are already a lot of pre-configured vendors that already have compatible appliances, like Cisco, F5, and Fortinet. Here<\/a> you can find the complete list. <\/p>\n\n\n\n In this article, we’ll see how to create a simple custom transparent IDS\/router appliance<\/strong> that you can adapt to your needs. We will use Linux, iptables, and Suricata<\/a>; building your custom appliance will also help you to understand what’s behind the scenes of pre-built solutions.<\/p>\n\n\n\n Before diving into the building phase, we will briefly describe how this solution works. <\/p>\n\n\n\n As we said, Gateway Load Balancers (GWLB) route all types of IP traffic (TCP, UDP, ICMP, GRE); the technology that makes this possible is the GENEVE protocol<\/strong>.<\/p>\n\n\n\n GENEVE is a new encapsulation protocol defined in RFC 8926<\/a>, a standard for different systems and vendors. Its acronym stands for Generic Network Virtualization Encapsulation. It encapsulates all traffic sending it in a virtual tunnel so that the underlying network is unaware of what’s inside. <\/p>\n\n\n\n This technology is used, for example, to extend and transport VLANS (or even VXLANs) across the Internet between different networks.<\/p>\n\n\n\n We want our setup to be fault<\/strong> tolerant and scalable<\/strong>. Like Application and Network Load Balancers, our Gateway Load Balancer can span multiple Availability Zones. We’ll also deploy our appliances using an Autoscaling Group so that we can add elasticity to our solution. <\/p>\n\n\n\n As shown in the figure below, we use NAT Gateways to simplify public IP management: some third-party external services can require a fixed set of source addresses. A Nat Gateway will ensure that, if the autoscaling adds an appliance in an availability zone, it still will use the same Elastic IP to access the Internet.<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n Let’s move on and start with some command-line and AWS Console sessions! <\/p>\n\n\n\n Before deploying our load balancer, we will need to create an AMI using Ubuntu 22.04; you can customize it later for your needs.<\/p>\n\n\n\n An EC2 instance can be used as a target for a GWLB if it can establish a GENEVE tunnel with it. Once the GENEVE tunnel is established, traffic can flow to it, and our Gateway Load Balancer will start distributing traffic. <\/p>\n\n\n\n For this reason, the first thing we need to do is to support a GENEVE tunnel on our custom appliance. Lucky for us, AWS already gives us a tool to ease our task<\/a> (You can also use the<\/p>\n\n\n\n Linux standard networking command to handle tunnel creation).<\/p>\n\n\n\n We will skip describing the instance creation and start with the tunnel handler compilation and installation. In this phase, we’ll install Suricata<\/strong>, an open-source network Intrusion Detection System, using the default configuration and updating its rules.<\/p>\n\n\n\n Our tunnel handler is ready. You will find an executable “gwlbtun” in the current directory; if you invoke it specifying the “-h” parameter, you should be able to see the help page.<\/p>\n\n\n\n Gwlbtun’s task is to establish the GENEVE connection with our GWLB; it also gives you the ability to specify a health check port that the target group will use, so you don’t have to use custom logic to implement one. <\/p>\n\n\n\n Additionally, it can run a script once the session is created or destroyed. We’ll take advantage of this and write a simple bash script that enables NAT (using iptables) and IP forwarding. Stopping the service will remove them.<\/p>\n\n\n\n Note: <\/strong> our instance will also need to disable a security feature called “source\/destination check”. This security feature blocks all traffic not originated or directed from or to the current instance as source or destination. As you’ll see, we need to add a role that enables the instance to set this flag by itself.<\/p>\n\n\n\n Place the following script in the<\/p>\n\n\n\n <\/p>\n\n\n\n directory and name it<\/p>\n\n\n\n <\/p>\n\n\n\n We now need to write a systemd unit that starts the handler, place it in<\/p>\n\n\n\n and give it a name. We will use aws-gwlb.service<\/p>\n\n\n\n Issue these commands to reload the configuration and enable the service. As this is only a template instance we don’t need to start it now.<\/p>\n\n\n\n You can now create an AMI and start with the Gateway Load balancer Creation.<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n We will not select any instance in the next step since we’ll use an Autoscaling Group.<\/p>\n\n\n\n Once the Target Group creation is completed, click on “Load Balancers”<\/strong>, add a new load balancer and Select “Gateway Load Balancer”. <\/p>\n\n\n\n The basic configuration is shared with the other load balancer types: you need to assign a name and select a VPC and its associated subnets. <\/p>\n\n\n\n You’ll find the target group we created in the “IP listener routing” section.<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n Once the Load Balancer creation finishes, define an endpoint service to use it. On the AWS console, click on VPC and go in the “Endpoint Services” section. The process is the same for endpoints based on Network Load Balancers (see here for details<\/a>).<\/p>\n\n\n\n Click on “Create endpoint Service”, give it a name, Select “Gateway” as type and then select the newly created load balancer:<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n Take note of the service name, you will need it when creating a new endpoint:<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n Click on “Endpoints”, “Create Endpoint”, select “Other endpoint services”, paste the service name we noted before and click on “Verify Service”.<\/p>\n\n\n\n Select the VPC and a subnet to place the endpoint (we use a subnet that is reachable from the Transit Gateway)<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n Repeat this step for the other subnets, and Don’t forget to accept the connections! <\/p>\n\n\n <\/p>\n\n\n\n Now our network configuration is complete! We only need to create the autoscaling group. Since this is a basic task, we will not cover it in this article. <\/p>\n\n\n\n Remember to create and include an instance profile in your launch template with this policy attached:<\/p>\n\n\n\n Note: this is a sample policy for our article. Remember that you may need to restrict its scope for security reasons.<\/p>\n\n\n\n Once you finish defining the autoscaling, you should see your instances running in the target group!<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n If you log into an instance, you will see that: <\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n 2. The service is up and running.<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n 3. Gwlbtun created two new network interfaces (gwi-* and gwo-*).<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n 4. Our firewall rules are present.<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n 5. Last but not least, Suricata will record network events.<\/p>\n\n\n\n <\/p>\n\n\n <\/p>\n\n\n\n You can install a squid proxy server on our template and use it as a transparent proxy to forward all logs to CloudWatch log for security investigation and auditing.<\/p>\n\n\n\n Hint: in \/etc\/squid\/squid.conf you will need to enable “transparent mode”, SSL bumping and insert the right NAT rules with iptables. With a Gateway Load Balancer, you can customize how outgoing traffic from your VPC is handled, gaining control and visibility using a highly available solution. <\/p>\n\n\n\n Now you should know what happens behind the scenes when you use a vendor appliance, Implementations may vary, but the key concepts and technology are in common. <\/p>\n\n\n\n Do you have other ideas or some additional unusual scenarios where a Gateway Load Balancer can help? Let us know in the comments!<\/p>\n\n\n\nHow Gateway Load Balancers work<\/h2>\n\n\n\n
Our Sample Architecture<\/h2>\n\n\n\n
<\/figure><\/div>\n\n\n
Install the tunnel handler and software<\/h2>\n\n\n\n
ip<\/code><\/pre>\n\n\n\n
apt update\napt install -y build-essential \"Development Tools\"\napt install -y cmake g++ suricata\nsnap install aws-cli --classic\nsuricata-update #update rules for suricata\ncd \/opt\ngit clone https:\/\/github.com\/aws-samples\/aws-gateway-load-balancer-tunnel-handler\ncd aws-gateway-load-balancer-tunnel-handler\ncmake .\nmake\n<\/code><\/pre>\n\n\n\n
root@ip-10-101-5-238:\/opt\/aws-gateway-load-balancer-tunnel-handler# .\/gwlbtun -h\nAWS Gateway Load Balancer Tunnel Handler\nUsage: .\/gwlbtun [options]\nExample: .\/gwlbtun\n\n -h Print this help\n -c FILE Command to execute when a new tunnel has been built. See below for arguments passed.\n -r FILE Command to execute when a tunnel times out and is about to be destroyed. See below for arguments passed.\n -t TIME Minimum time in seconds between last packet seen and to consider the tunnel timed out. Set to 0 (the default) to never time out tunnels.\n Note the actual time between last packet and the destroy call may be longer than this time.\n -p PORT Listen to TCP port PORT and provide a health status report on it.\n -s Only return simple health check status (only the HTTP response code), instead of detailed statistics.\n -d Enable debugging output.\n -x Enable dumping the hex payload of packets being processed.\n\n---------------------------------------------------------------------------------------------------------\nTunnel command arguments:\nThe commands will be called with the following arguments:\n1: The string 'CREATE' or 'DESTROY', depending on which operation is occurring.\n2: The interface name of the ingress interface (gwi-<X>).\n3: The interface name of the egress interface (gwo-<X>). Packets can be sent out via in the ingress\n as well, but having two different interfaces makes routing and iptables easier.\n4: The GWLBE ENI ID in base 16 (e.g. '2b8ee1d4db0c51c4') associated with this tunnel.\n\nThe <X> in the interface name is replaced with the base 60 encoded ENI ID (to fit inside the 15 character\ndevice name limit).<\/code><\/pre>\n\n\n\n
\/opt\/aws-gateway-load-balancer-tunnel-handler<\/code><\/pre>\n\n\n\n
tunnel-handler.sh<\/code><\/pre>\n\n\n\n
#!\/bin\/bash\n\n# Note: This requires this instance to have Source\/Dest check disabled; we need to assign a role to the ec2 instance to enable and disable it\n\n\necho \"Running tunnel handler script... \"\necho Mode is $1, In Int is $2, Out Int is $3, ENI is $4\n\niptables -F\niptables -t nat -F\nINSTANCE_ID=$(curl 169.254.169.254\/latest\/meta-data\/instance-id\n\ncase $1 in\n \tCREATE)\n\t\t\techo \"Disabling source and destination check.\"\n\t\t\taws ec2 modify-instance-attribute --instance-id=$INSTANCE_ID --source-dest-check\n\n \techo \"Setting up NAT and IP FORWARD\"\n \tiptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE\n \tiptables -A FORWARD -i $2 -o $2 -j ACCEPT\n \techo 1 > \/proc\/sys\/net\/ipv4\/ip_forward\n \techo 0 > \/proc\/sys\/net\/ipv4\/conf\/all\/rp_filter\n \techo 0 > \/proc\/sys\/net\/ipv4\/conf\/$2\/rp_filter\n \t;;\n \tDESTROY)\n\t\t\techo \"Enabling source and destination check.\"\n\t\t\taws ec2 modify-instance-attribute --instance-id=$INSTANCE_ID --no-source-dest-check\n \techo \"Removing IP FORWARD\"\n \techo 0 > \/proc\/sys\/net\/ipv4\/ip_forward\n \techo 1 > \/proc\/sys\/net\/ipv4\/conf\/all\/rp_filter\n \techo 1 > \/proc\/sys\/net\/ipv4\/conf\/$2\/rp_filter\n \t;;\n \t*)\n \techo \"invalid action.\"\n \texit 1\n \t;;\nesac<\/code><\/pre>\n\n\n\n
\/lib\/systemd\/system<\/code><\/pre>\n\n\n\n
[Unit]\nDescription=AWS GWLB Tunnel Handler\nAfter=network.target\n\n[Service] \nExecStart=\/opt\/aws-gateway-load-balancer-tunnel-handler\/gwlbtun -c \/opt\/aws-gateway-load-balancer-tunnel-handler\/tunnel-handler.sh -r \/opt\/aws-gateway-load-balancer-tunnel-handler\/tunnel-handler.sh -p 80\nRestart=always\nRestartSec=5s\n\n[Install]\nWantedBy=multi-user.target\nAlias=aws-gwlb<\/code><\/pre>\n\n\n\n
systemctl daemon-reload\nsystemctl enable aws-gwlb<\/code><\/pre>\n\n\n\n
Load Balancer Configuration<\/h2>\n\n\n\n
First, create a Target Group, click on “Target Groups<\/strong>“, and create a new one.
Select “Instances” for the target type, give it a name, and select “GENEVE<\/strong>” as protocol. We will use port 80 as health check target because we told our tunnel handler to use that port (the “-p 80″ command line switch”).<\/p>\n\n\n\n<\/figure><\/div>\n\n\n
<\/figure><\/div>\n\n\n
<\/figure><\/div>\n\n\n
<\/figure><\/div>\n\n\n
<\/figure><\/div>\n\n\n
When our endpoint is ready, we can modify our route table to use it and direct our traffic to the gateway load balancer. Select “Gateway Load Balancer Endpoint<\/strong>” as the target:<\/p>\n\n\n\n<\/figure><\/div>\n\n\n
{\n \"Sid\": \"Allow Source-Dest check modification\",\n \"Effect\": \"Allow\",\n \"Action\": \"ec2:ModifyInstanceAttribute\",\n \"Resource\": \"*\"\n}<\/code><\/pre>\n\n\n\n
<\/figure><\/div>\n\n\n
<\/figure><\/div>\n\n\n
<\/figure><\/div>\n\n\n
<\/figure><\/div>\n\n\n
<\/figure><\/div>\n\n\n
<\/figure><\/div>\n\n\n
Next steps<\/h2>\n\n\n\n
You can filter outgoing traffic if you refine the firewall script (or use a graphical interface like EasyWall<\/a>.<\/p>\n\n\n\nTo conclude<\/h2>\n\n\n\n
\n\n\n\nAbout Proud2beCloud<\/h4>\n\n\n\n