Posts in Aws (5 found)
DuckTyped 6 months ago

An illustrated guide to route tables

Hi reader! This is part of a series on AWS Networking . The story so far: we've learned about VPCs . We've learned about subnets . We created a subnet and an internet gateway . Now we want to connect our subnet to the internet gateway. In order to do that, we need to learn about CIDR notation and route tables. This chapter is on route tables (“root tables”?). Make a new route table Add a rule to route to your internet gateway Associate this route table with a subnet Code for doing this Longer explanation: Someone wants to connect to your server. A packet comes in, with a destination IP address, and says "help I'm trying to go to 175.88.11.12". How do we know where the packet needs to go? Another packet comes up and says "help I'm trying to get to 142.250.191.110". This packet actually wants to go out to the internet. How do we direct this packet? The answer to both questions is a route table! A route table will map a destination to a target. Route tables are set up so given a destination, the route table tells you the next step to take . It's just like if you're trying to go to a city: In this example, the route table would look like this The destination you're trying to get to is Mumbai, and the next step you need to go to is the airport. Nuance! The route table doesn't tell the packet how to get to its final destination. It just tells the packet the NEXT STEP to take. We want to connect a subnet to an internet gateway. The route table for that might look like this: Lets break down what this table is saying. Suppose a packet comes in with a destination IP address. That IP address can be within your VPC, in which case the route table should route it locally The route table tells the packet where to go next. If it's any other IP address, it should route it to your internet gateway What does the route table for the logic above look like? Well, to start we need two routes: If our EC2 instance wants to hit an IP address inside our VPC, we route that request locally inside the VPC if it wants to hit any other IP address , we route that request to the internet, through the internet gateway Here's what the route table for the logic above would look like: You’ll notice we are using CIDR notation to specify a range of IP addresses (quick tutorial on CIDR here ). The first rule says, any IP address that starts with should get routed locally. The second rule says, all IP addresses should get routed to the internet gateway. You can add routes for specific IP addresses, but typically you will create routes for CIDR ranges. The route table in our example had two routes Notice that one of the routes matches any IP address while the other one matches only IP addresses in your VPC So if a packet comes in with a local IP address, technically it matches both routes. Where will it go? Route tables will always try to match the most specific route . Psst… how is the matching done? Using something called a Genmask. This isn't need-to-know information, so I won't describe it here, but now you know the word and can look it up if you want to. Route tables are created at the VPC level. This means VPCs can have many route tables. Every VPC comes with a route table, called the main route table . This route table will have a route by default that maps your VPC's CIDR range to the target. Typically, you want to leave this as is. All subnets are associated with the main route table by default. Suppose you have two subnets, both private. You're trying to make one of them public, i.e. giving it a connection to the internet. You do this by attaching it to an internet gateway. To do that , you need to add a route to your route table. Suppose you add a route like this to your main route table: This works, and one of your subnets is now public. Unfortunately, all of your subnets are now public! Since you made this change on the main route table , and all subnets are associated with the main route table by default, all your subnets are now public. To make a subnet public, you instead want to make a new route table and associate your subnet with that route table. But there's a gotcha here. Subnets can only be associated with one route table at a time , so now your subnet will no longer be associated with the main route table. Remember that the main route table had that route for local traffic. Now your public subnet can no longer handle local traffic! If you want the subnet to handle local traffic, you'll have to add a route to your new route table to handle local traffic . (Excuse the lack of syntax highlighting — Substack doesn’t have that) Here is the Terraform code for everything we have learned so far, including making a VPC, subnet, etc: Github gist If you are SSHed into an EC2 instance, you can see the route table by using the command. You can also do , which just calls in the background. Phew! You finally have a route in place, and your subnet is connected to the internet gateway. Here's what we have so far: Bullet points To connect a subnet to the internet, you need a route. Routes are defined as rules in a route table. A route table matches a destination to a target. Destinations are usually specified as CIDR ranges. Route tables are created on the VPC level. Each VPC comes with a default route table called the main route table. You probably want to create a new route table and associate it with a subnet. Subnets can only be associated with one route table at a time. Your subnet is now connected to the internet, but you still need to set up your security rules to allow traffic in. We’ll see that next! Chapter 5: Security groups Thanks for reading DuckTyped! Subscribe for free to receive new posts and support my work. Make a new route table Add a rule to route to your internet gateway Associate this route table with a subnet In this example, the route table would look like this The destination you're trying to get to is Mumbai, and the next step you need to go to is the airport. Nuance! The route table doesn't tell the packet how to get to its final destination. It just tells the packet the NEXT STEP to take. We want to connect a subnet to an internet gateway. The route table for that might look like this: Lets break down what this table is saying. Local or internet? Suppose a packet comes in with a destination IP address. That IP address can be within your VPC, in which case the route table should route it locally The route table tells the packet where to go next. If it's any other IP address, it should route it to your internet gateway What does the route table for the logic above look like? Well, to start we need two routes: If our EC2 instance wants to hit an IP address inside our VPC, we route that request locally inside the VPC if it wants to hit any other IP address , we route that request to the internet, through the internet gateway Here's what the route table for the logic above would look like: You’ll notice we are using CIDR notation to specify a range of IP addresses (quick tutorial on CIDR here ). The first rule says, any IP address that starts with should get routed locally. The second rule says, all IP addresses should get routed to the internet gateway. You can add routes for specific IP addresses, but typically you will create routes for CIDR ranges. Route tables pick the most specific route that matches The route table in our example had two routes Notice that one of the routes matches any IP address while the other one matches only IP addresses in your VPC So if a packet comes in with a local IP address, technically it matches both routes. Where will it go? Route tables will always try to match the most specific route . Psst… how is the matching done? Using something called a Genmask. This isn't need-to-know information, so I won't describe it here, but now you know the word and can look it up if you want to. The main route table Route tables are created at the VPC level. This means VPCs can have many route tables. Every VPC comes with a route table, called the main route table . This route table will have a route by default that maps your VPC's CIDR range to the target. Typically, you want to leave this as is. Subnets and route tables All subnets are associated with the main route table by default. Suppose you have two subnets, both private. You're trying to make one of them public, i.e. giving it a connection to the internet. You do this by attaching it to an internet gateway. To do that , you need to add a route to your route table. Suppose you add a route like this to your main route table: This works, and one of your subnets is now public. Unfortunately, all of your subnets are now public! Since you made this change on the main route table , and all subnets are associated with the main route table by default, all your subnets are now public. To make a subnet public, you instead want to make a new route table and associate your subnet with that route table. But there's a gotcha here. Subnets can only be associated with one route table at a time , so now your subnet will no longer be associated with the main route table. Remember that the main route table had that route for local traffic. Now your public subnet can no longer handle local traffic! If you want the subnet to handle local traffic, you'll have to add a route to your new route table to handle local traffic . Terraform (Excuse the lack of syntax highlighting — Substack doesn’t have that) Here is the Terraform code for everything we have learned so far, including making a VPC, subnet, etc: Github gist Bonus command line tip! If you are SSHed into an EC2 instance, you can see the route table by using the command. You can also do , which just calls in the background. Summary Phew! You finally have a route in place, and your subnet is connected to the internet gateway. Here's what we have so far: Bullet points To connect a subnet to the internet, you need a route. Routes are defined as rules in a route table. A route table matches a destination to a target. Destinations are usually specified as CIDR ranges. Route tables are created on the VPC level. Each VPC comes with a default route table called the main route table. You probably want to create a new route table and associate it with a subnet. Subnets can only be associated with one route table at a time.

aws AWS tutorial Tutorial cloud Cloud
0 views
DuckTyped 8 months ago

An illustrated guide to Amazon VPCs: bonus content

This post contains some bonus info for An illustrated guide to Amazon VPCs . One reader added some fascinating insights: Security groups were the original solution to the second problem (that everyone's servers were in the same network). The address conflict problem transformed into another problem after VPCs. After everybody got default VPCs, companies went from conflicting with each other to conflicting with themselves. Now they have a bunch of instances with the same IPs, in different VPCs, that still need to talk to each other… a problem that needed double-sided NAT. A big part of the first problem (IP address conflicts) was actually that AWS was about to run out of IP addresses to issue to customers. Back then, every instance got a public address because there was no other way to ssh into your instance. Because public addresses are so expensive and limited, this was a scaling cliff for AWS, and passed on as an unnecessary cost to customers. VPCs let you use a single bastion with a public IP to access the rest of your hosts.

aws AWS cloud Cloud
0 views
DuckTyped 9 months ago

Old man yells at cloud: a mini-book on AWS networking

If you have ever tried to put an app on AWS, you know that you first need to absorb a ton of knowledge about AWS networking. Before you set up that first EC2 instance, you need to know terms like VPCs, subnets, security groups, and internet gateways. You need to remember how CIDR notation works, and that networking class you took suddenly feels like a long time ago. Without a solid understanding of AWS networking, you may be able to start up an EC2 instance, but you will have a hard time getting it connected to the internet. This series of posts will get you up to speed on AWS networking, so you can get an app up on AWS. Every post is meant to be easy to understand, and spends time really explaining the concepts, so you understand how things work, and can adapt them to fit your use case. Each post is full of illustrations that I hope will brighten your journey as you learn AWS networking. Good luck and happy reading! Read Chapter 1: VPCs Route tables Security groups AWS in Terraform Thanks for reading DuckTyped! Subscribe for free to receive new posts and support my work. Route tables Security groups AWS in Terraform

tutorial Tutorial cloud Cloud aws AWS
0 views
Humble Thoughts 1 years ago

AWS EFS Configuration failure

I wanted to share a quick story about setting up an AWS EFS instance. If you're reading this, I assume you're either in the process of configuring one or have encountered a similar issue. Here's what happened: You successfully set up an EFS, configured access points for it, and mounted it to an EC2 instance or ECS service cluster. You thought everything was going smoothly because you successfully set up NAT networking. However, when you tried to write data to the filesystem, it kept failing with a exception. This happens due to misconfiguration of the access point permissions: the directory created in the EFS doesn't imply permissions for the user trying to reach the data on disk. For example, imagine an access point that was made with the configuration to be owned by the uid and gid with the permissions set to : Initial Access point configuration In this case, trying to reach the data under a non-root user with a different uid and gid from the mentioned will fail. So, first things first, you should ensure the permissions of the AP are set to allow the user to work with the data. Though, here's the catch. Imagine you've tried the configuration above and understood it doesn't work. After that, you dive into the container image code or the application configuration and learn the user working with the drive is . You recreate the AP as there's no way to reconfigure those settings, but it still doesn't work... This is where it's important to remember that even if you reconfigured the AP settings, they still won't recreate the FS directory nor reconfigure its permissions for you. Reconfigured AP You end up trying to write to the directory owned by the user under the user, so it fails. Even if you set permissions to and recreate an AP with it, it won't change anything. So, you have two options now: either manually adjust the attributes of the directory on FS to fit your new configuration ( , ) or create a new directory with the correct settings.

devops DevOps cloud Cloud aws AWS
0 views
Karan Sharma 2 years ago

Storing AWS Pinpoint Logs

At $dayjob, we use AWS Pinpoint to send out SMS to our customers. We’ve also written a detailed blog post on how we use Clickhouse + vector stack for our logging needs. We additionally wanted to store the delivery logs generated by the Pinpoint service. But like with anything else in AWS, even simpler tasks like these usually tend to piggyback on other counterparts of AWS - in this case, it happens to be AWS Kinesis. All the delivery logs which contain metadata about SMS delivery are streamed to Kinesis. Our setup involves configuring Pinpoint with Amazon Kinesis Data Firehose stream. Firehose is an ETL service that helps stream events to other persistent stores. Firehose supports multiple such output sinks and in our case we use sink. This is what the flow looks like: On the HTTP server side, we used ’s aws_kinesis_firehose source. Compared to just using http source, here are the differences I found: Has first-class support for access_key. AWS Kinesis can be configured to send access_key which comes as the value header in the HTTP request. This means that the request which contains an invalid access key will be rejected at the source itself. However, in the source, I couldn’t find a way to drop events at the source level. It is required to use a VRL transformer to check whether is present in the headers and do a value comparison with our key. Has native support for decoding the payload. This one’s pretty useful and saved me a lot of VRL transformer rules that I would have otherwise written with the source. So, basically, this is how the server receives the payload: The value of the payload is a base64 encoded value of the JSON Object of an SMS event. However, the source is smart enough and automagically decodes this list of records and their values into individual events. This is how the final event looks like when using source: This makes it straightforward because now we just have to parse the JSON inside the key and do transformations on that object. If it was source, then I’d to loop over the records structure and figure out how to split them as individual events for the rest of the Vector pipeline… which would have been messy to say the least. Here’s the vector config so far: Now that we have a pipeline which sends and receives data, we can process the events and transform them into a schema that is more desirable. Since we require the events to be queryable in a Clickhouse DB, this is the schema we have: To achieve the above format, we can use VRL to parse and format our SMS events: Plugging this, we have a clean JSON object for each SMS event. The only thing now we need to add is an output sink to Clickhouse: Perfect! On running this pipeline with we can see the consumption the records Hope this short post was useful if you’ve to do anything similar! Has first-class support for access_key. AWS Kinesis can be configured to send access_key which comes as the value header in the HTTP request. This means that the request which contains an invalid access key will be rejected at the source itself. However, in the source, I couldn’t find a way to drop events at the source level. It is required to use a VRL transformer to check whether is present in the headers and do a value comparison with our key. Has native support for decoding the payload. This one’s pretty useful and saved me a lot of VRL transformer rules that I would have otherwise written with the source. So, basically, this is how the server receives the payload: The value of the payload is a base64 encoded value of the JSON Object of an SMS event. However, the source is smart enough and automagically decodes this list of records and their values into individual events. This is how the final event looks like when using source: This makes it straightforward because now we just have to parse the JSON inside the key and do transformations on that object. If it was source, then I’d to loop over the records structure and figure out how to split them as individual events for the rest of the Vector pipeline… which would have been messy to say the least.

aws AWS json JSON devops DevOps
0 views