DuckTyped 1 weeks ago

An Illustrated Introduction to Linear Algebra

This post assumes you know algebra, but no linear algebra. Lets dive in. There are two big ideas I want to introduce in the first chapter: Gaussian elimination, (which is not strictly a linear algebra thing, and has been around for years before linear algebra came along), and row picture versus column picture, which is a linear algebra thing. Let’s say you have a bunch of nickels and pennies, and you want to know how many of each do you need to have 23 cents . You could write that as an equation that looks like this: is the number of nickels you need, is the number of pennies you need. And you need to figure out the and values that would make the left-hand side work out to 23. And this one is pretty easy, you can just work it out yourself. You’d need four nickels and three pennies. So is four, is three. This kind of equation is called a linear equation . And that’s because when you plot this equation, everything is flat and smooth. There are no curves or holes. There isn’t a in the equation for example to make it curved. Linear equations are great because they’re much easier to work with than curved equations. Aside: Another solution for the above is 23 pennies. Or -4 nickels + 43 pennies. The point is you have two variables (x and y for nickels and pennies), and you are trying to combine them in different ways to hit one number . The trouble starts when you have two variables, and you need to combine them in different ways to hit two different numbers . That’s when Gaussian elimination comes in. In what world would you have to hit two different numbers? Does that seem outlandish? It’s actually very common! Read on for an example. Now let’s look at a different example. In the last one we were trying to make 23 cents with nickels and pennies. Here we have two foods. One is milk, the other is bread. They both have some macros in terms of carbs and protein: and now we want to figure out how many of each we need to eat to hit this target of 5 carbs and 7 protein. This is a very similar question to the one we just asked with nickels and pennies, except instead of one equation, we have two equations: Again we have an and a . Lets find their values. To solve these kinds of questions, we usually use Gaussian elimination . If you’ve never used Gaussian elimination, strap in. Step one is to rewrite this as a set of two equations: Now you subtract multiples of one equation from another to try to narrow down the value of one variable. Lets double that second equation: See how we have a and a now? Now we can add the two equations together to eliminate : We’re left with one equation and one variable. We can solve for : Aha, we know . Now we can plug that into one of the equations to find . We plug that in to one of the equations and find out that equals 1, and there we have answer: three milks, one bread, is what we need. This method is called Gaussian elimination, even though it was not discovered by Gauss. If you haven’t seen Gaussian elimination, congratulations, you learned a big idea! Gaussian elimination is something we will talk about more. It’s part of what makes linear algebra useful. We can also find the solution by drawing pictures. Let’s see how that works. Let’s plot one of these lines. First, we need to rewrite the equations in terms of . Reminder: first equation is for carbs, second for protein. x is number of milks, y is number of breads. Now let’s plot the graph for the first equation. Now, what does this line represent? It’s all the combinations of bread and milk that you can have to get exactly five carbs: So you can eat no milk and two-and-a-half breads, or two milks and one-and-a-half breads, or five milks and no bread, to get to exactly five carbs. All of those combinations would mean you have eaten exactly five carbs. You can pick any point that sits on this line to get to your goal of eating five carbs. Note: You can see the line goes into the negative as well. Technically, 5 breads and -5 milks will give you 5 carbs as well, but you can’t drink negative milks. For these examples, let’s assume only positive numbers for the variables. Now, let’s plot the other one. This is the same thing, but for protein. If you eat any of these combinations, you’ll have met the protein goal: You can pick a point that sits on the first line to meet the carb goal. You can pick a point that sits on the second line to meet the protein goal. But you need a point that sits on both lines to hit both goals. How would a point sit on both lines? Well, it would be where the lines cross . Since these are straight lines, the lines cross only once, which makes sense because there’s only a single milk and bread combo that would get you to exactly five grams of carbs and seven grams of protein. Now we plot the lines together, see where they intersect, and that’s our answer: Bam! We just found the solution using pictures. So that’s a quick intro to Gaussian elimination. But you don’t need linear algebra to do Gaussian elimination. This is a technique that has been around for 2,000 years. It was discovered in Asia, it was rediscovered in Europe, I think in the 1600s or something, and no one was really talking about “linear algebra”. This trick is just very useful. That’s the first big idea you learned. You can stop there if you want. You can practice doing this sort of elimination. It’s a very common and useful thing. What we just saw is called the “row picture”. Now I want to show you the column picture. I’m going to introduce a new idea, which is: instead of writing this series of equations, what if we write just one equation? Remember how we had one equation for the nickels and pennies question? What if we write one like that for food? Not a system of equations, just a single equation? What do you think that would look like? Something like this: It’s an equation where the coefficients aren’t numbers, they’re an “array” of numbers. The big idea here is: what if we have a linear equation, but instead of numbers, we have arrays of numbers? What if we treat , the way we treat a number? Can that actually work? If so, it is pretty revolutionary. Our whole lives we have been looking at just numbers, and now we’re saying, what if we look at arrays of numbers instead? Let’s see how it could work in our food example. What if the coefficients are an array of numbers? Well, this way of thinking is actually kind of intuitive. You might find it even more intuitive than the system of equations version. Each of these coefficients are called vectors . If you’re coming from computer science, you can kind of think of a vector as an array of numbers (i.e. the order matters). Lets see how we can use vectors to find a solution to the bread and milk question. Yeah, we can graph vectors . We can graph them either as a point, like I’ve done for the target vector here, or as an arrow, which is what I’ve done with the vector for bread and the vector for milk: Use the two numbers in the vector as the x and y coordinates. That is another big idea here: We always think of a set of coordinates giving a point, but you can think of vectors as an arrow instead of just a point . Now what we’re asking is how much milk and how much bread do we need, to get to that point? This is a pretty simple question. It’s simple enough that we can actually see it. Let me add some milks: And let me add a bread. Bingo bango, we’re at the point: Yeah, we literally add them on, visually. I personally find this more intuitive. I think the system of equations picture can confuse me sometimes, because the initial question was, “how much bread and how much milk should I eat?” The vector way, you see it in terms of breads and milks. The row way, you see it as one of the lines is the carbs, the other line is the protein, and the x and y axes are the amount of bread, which results in the same thing, but it’s a little more roundabout, a little more abstract. This one is very direct. We just saw that we can graph vectors too. Graphing it works differently from graphing the rows, but there is a graph we can make, and it works, which is pretty cool. What about the algebra way? Here is the equation again: Since we already know the answer, I’ll just plug that in: Now, the question is how does the left side equal the right side? The first question is how do you define this multiplication? Well, in linear algebra, it’s defined as, if you multiply a scalar by a vector, you just multiply it by each number in that vector: Now you are left with two vectors. How do you add two vectors? Well, in the linear algebra you just add the individual elements of each vector: And you end up with the answer. Congratulations, you’ve just had your first taste of linear algebra. It’s a pretty big step, right? Instead of numbers, we’re working with arrays of numbers. In future chapters, we will see why this is so powerful. That’s the first big concept of linear algebra: row picture vs column picture. Finally, I’ll just leave you with this last teaser, which is: how would you write these two equations in matrix notation? Like this: This is the exact same thing as before. You can write it as scalars times columns, as we had done before: or you can write it as a matrix times a vector, as above. Either one works. Matrices are a big part of linear algebra. But before we talk about matrices, we will talk about the dot product, which is coming up next. Check out Gilbert Strang’s lectures on linear algebra on YouTube . Thanks for reading DuckTyped! Subscribe for free to receive new posts and support my work. P.S. Want more art? Check out my Instagram . Let’s say you have a bunch of nickels and pennies, and you want to know how many of each do you need to have 23 cents . You could write that as an equation that looks like this: is the number of nickels you need, is the number of pennies you need. And you need to figure out the and values that would make the left-hand side work out to 23. And this one is pretty easy, you can just work it out yourself. You’d need four nickels and three pennies. So is four, is three. This kind of equation is called a linear equation . And that’s because when you plot this equation, everything is flat and smooth. There are no curves or holes. There isn’t a in the equation for example to make it curved. Linear equations are great because they’re much easier to work with than curved equations. Aside: Another solution for the above is 23 pennies. Or -4 nickels + 43 pennies. The point is you have two variables (x and y for nickels and pennies), and you are trying to combine them in different ways to hit one number . The trouble starts when you have two variables, and you need to combine them in different ways to hit two different numbers . That’s when Gaussian elimination comes in. In what world would you have to hit two different numbers? Does that seem outlandish? It’s actually very common! Read on for an example. Food example Now let’s look at a different example. In the last one we were trying to make 23 cents with nickels and pennies. Here we have two foods. One is milk, the other is bread. They both have some macros in terms of carbs and protein: and now we want to figure out how many of each we need to eat to hit this target of 5 carbs and 7 protein. This is a very similar question to the one we just asked with nickels and pennies, except instead of one equation, we have two equations: Again we have an and a . Lets find their values. To solve these kinds of questions, we usually use Gaussian elimination . If you’ve never used Gaussian elimination, strap in. Gaussian elimination Step one is to rewrite this as a set of two equations: Now you subtract multiples of one equation from another to try to narrow down the value of one variable. Lets double that second equation: See how we have a and a now? Now we can add the two equations together to eliminate : We’re left with one equation and one variable. We can solve for : Aha, we know . Now we can plug that into one of the equations to find . We plug that in to one of the equations and find out that equals 1, and there we have answer: three milks, one bread, is what we need. This method is called Gaussian elimination, even though it was not discovered by Gauss. If you haven’t seen Gaussian elimination, congratulations, you learned a big idea! Gaussian elimination is something we will talk about more. It’s part of what makes linear algebra useful. We can also find the solution by drawing pictures. Let’s see how that works. Picture version Let’s plot one of these lines. First, we need to rewrite the equations in terms of . Reminder: first equation is for carbs, second for protein. x is number of milks, y is number of breads. Now let’s plot the graph for the first equation. Now, what does this line represent? It’s all the combinations of bread and milk that you can have to get exactly five carbs: So you can eat no milk and two-and-a-half breads, or two milks and one-and-a-half breads, or five milks and no bread, to get to exactly five carbs. All of those combinations would mean you have eaten exactly five carbs. You can pick any point that sits on this line to get to your goal of eating five carbs. Note: You can see the line goes into the negative as well. Technically, 5 breads and -5 milks will give you 5 carbs as well, but you can’t drink negative milks. For these examples, let’s assume only positive numbers for the variables. Now, let’s plot the other one. This is the same thing, but for protein. If you eat any of these combinations, you’ll have met the protein goal: You can pick a point that sits on the first line to meet the carb goal. You can pick a point that sits on the second line to meet the protein goal. But you need a point that sits on both lines to hit both goals. How would a point sit on both lines? Well, it would be where the lines cross . Since these are straight lines, the lines cross only once, which makes sense because there’s only a single milk and bread combo that would get you to exactly five grams of carbs and seven grams of protein. Now we plot the lines together, see where they intersect, and that’s our answer: Bam! We just found the solution using pictures. So that’s a quick intro to Gaussian elimination. But you don’t need linear algebra to do Gaussian elimination. This is a technique that has been around for 2,000 years. It was discovered in Asia, it was rediscovered in Europe, I think in the 1600s or something, and no one was really talking about “linear algebra”. This trick is just very useful. That’s the first big idea you learned. You can stop there if you want. You can practice doing this sort of elimination. It’s a very common and useful thing. The column picture What we just saw is called the “row picture”. Now I want to show you the column picture. I’m going to introduce a new idea, which is: instead of writing this series of equations, what if we write just one equation? Remember how we had one equation for the nickels and pennies question? What if we write one like that for food? Not a system of equations, just a single equation? What do you think that would look like? Something like this: It’s an equation where the coefficients aren’t numbers, they’re an “array” of numbers. The big idea here is: what if we have a linear equation, but instead of numbers, we have arrays of numbers? What if we treat , the way we treat a number? Can that actually work? If so, it is pretty revolutionary. Our whole lives we have been looking at just numbers, and now we’re saying, what if we look at arrays of numbers instead? Let’s see how it could work in our food example. What if the coefficients are an array of numbers? Well, this way of thinking is actually kind of intuitive. You might find it even more intuitive than the system of equations version. Each of these coefficients are called vectors . If you’re coming from computer science, you can kind of think of a vector as an array of numbers (i.e. the order matters). Lets see how we can use vectors to find a solution to the bread and milk question. Step one: graph the vectors. Yeah, we can graph vectors . We can graph them either as a point, like I’ve done for the target vector here, or as an arrow, which is what I’ve done with the vector for bread and the vector for milk: Use the two numbers in the vector as the x and y coordinates. That is another big idea here: We always think of a set of coordinates giving a point, but you can think of vectors as an arrow instead of just a point . Now what we’re asking is how much milk and how much bread do we need, to get to that point? This is a pretty simple question. It’s simple enough that we can actually see it. Let me add some milks: And let me add a bread. Bingo bango, we’re at the point: Yeah, we literally add them on, visually. I personally find this more intuitive. I think the system of equations picture can confuse me sometimes, because the initial question was, “how much bread and how much milk should I eat?” The vector way, you see it in terms of breads and milks. The row way, you see it as one of the lines is the carbs, the other line is the protein, and the x and y axes are the amount of bread, which results in the same thing, but it’s a little more roundabout, a little more abstract. This one is very direct. The algebra way We just saw that we can graph vectors too. Graphing it works differently from graphing the rows, but there is a graph we can make, and it works, which is pretty cool. What about the algebra way? Here is the equation again: Since we already know the answer, I’ll just plug that in: Now, the question is how does the left side equal the right side? The first question is how do you define this multiplication? Well, in linear algebra, it’s defined as, if you multiply a scalar by a vector, you just multiply it by each number in that vector: Now you are left with two vectors. How do you add two vectors? Well, in the linear algebra you just add the individual elements of each vector: And you end up with the answer. Congratulations, you’ve just had your first taste of linear algebra. It’s a pretty big step, right? Instead of numbers, we’re working with arrays of numbers. In future chapters, we will see why this is so powerful. That’s the first big concept of linear algebra: row picture vs column picture. Finally, I’ll just leave you with this last teaser, which is: how would you write these two equations in matrix notation? Like this: This is the exact same thing as before. You can write it as scalars times columns, as we had done before: or you can write it as a matrix times a vector, as above. Either one works. Matrices are a big part of linear algebra. But before we talk about matrices, we will talk about the dot product, which is coming up next. Additional reading Check out Gilbert Strang’s lectures on linear algebra on YouTube .

Programming Data Analysis Tutorial

▲ 0

0 views

DuckTyped 1 months ago

An Illustrated Guide to OAuth

OAuth was first introduced in 2007. It was created at Twitter because Twitter wanted a way to allow third-party apps to post tweets on users' behalf. Take a second to imagine designing something like that today. How would you do it? One way would just be to ask the user for their username and password. So you create an unofficial Twitter client, and present the user a login screen that says "log in with Twitter". The user does so, but instead of logging into Twitter, they're actually sending their data to you, this third-party service which logs into Twitter for them. This is bad for a lot of reasons. Even if you trust a third-party app, what if they don't store your password correctly and someone steals it? You should never give your password to a third-party website like this. Another way you might be thinking is, what about API keys? Because you're hitting Twitter's API to post data for a user, and for an API , you use API keys . But API keys are general. What you need is an API key specific to a user. To solve these problems, OAuth was created. You'll see how it solves all these problems, but the crux of OAuth is an access token , which is sort of like an API key for a specific user. An app gets an access token, and then they can use that to take actions on the user's behalf, or access data for a user. OAuth can be used in a lot of different ways, one of the reasons it is so hard to understand. In this post, we’re going to look at a typical OAuth flow. The example I'm going to use is YNAB. If you haven't used it, YNAB is like a paid version of Mint. You connect it to a bank account, and then it pulls all your transactions from that account, and shows them to you with very pretty charts. You can categorize your spending, and then it tells you for example, hey, you're spending too much on groceries. It helps you manage your finances. So, I want to use YNAB, and I want to connect it to Chase Bank, but I don't want to give it my Chase password. So instead, I'm going to use OAuth. Let's look at the flow first, and then let's understand what's going on. We're actually going to look at the flow twice , because I think you need to look through an OAuth flow at least two times to understand what's going on. So to start, I'm at YNAB, and I want to connect Chase as a source. The OAuth flow looks like this: YNAB redirects me to Chase. At Chase, I log in with my username and password. Chase shows me a screen saying "YNAB wants to connect to Chase. Pick what accounts you want to give YNAB access to". It'll show me a list of all my accounts. Let's say I pick just my checking account, to give YNAB read access to this account, and hit OK. From Chase, I'm redirected back to YNAB, and now, magically, YNAB is connected to Chase. This is the experience from a user's perspective. But what happened there? What magic happened in the background, so that YNAB somehow has access to my data on Chase? Remember, the end goal of OAuth is for YNAB to end up with an access token , so it can access my data from Chase. Somehow, as I went through this flow, YNAB ended up with an access token. I'll spoil the surprise by telling you how it got the access token, and then I'll walk you through what happened in more detail. How does Chase give YNAB the access token? When you were redirected from Chase back to YNAB, Chase could have just added the access token in the URL. It could have redirected you back to a URL like this: and then YNAB would be able to get the access token. An access token is supposed to be secret, but URLs can end up in your browser's history or some server logs, in which case it's easy for anyone to see your access token. So Chase could technically redirect you back to YNAB with the access token in the URL , and then YNAB would have the access token. End of OAuth flow. But we don’t do it this way, because sending an access token in the URL is not secure. When you were redirected from Chase back to YNAB, Chase sent YNAB an authorization code in the URL. An authorization code is not an access token! Chase sends YNAB an authorization code, and YNAB exchanges the authorization code for an access token . It does this by making a backend request to Chase, a backend POST request over HTTPS, which means no one can see the access token. And then YNAB has the access token. End of OAuth flow. OAuth success. Let's talk about what we just saw. At a high level, there are two parts to an OAuth flow. The first is the user consent flow , which is where you, the user, log in and pick what to give access to. This is a critical part of OAuth, because in OAuth, we always want the user to be actively involved and in control. The other part is the authorization code flow . This is the flow where YNAB actually gets this access token . Let's talk about more details of exactly how this works. And let's also talk about some terminology, because OAuth has very specific terminology. Instead of user, we say resource owner . Instead of app, we say OAuth client or OAuth app . The server where you log in is called the authorization server . The server where you get user data from is called the resource server (This could be the same as the authorization server). On the authorization server, when the user picks what's allowed, those are called scopes . I'll try to use that terminology, because you'll need to get familiar with it if you're going to read more OAuth documentation. So let’s look at this high level again, with the new terms. You have OAuth clients. An OAuth client wants to access data on a resource server, and the data belongs to the resource owner. To do that, the OAuth client redirects to the authorization server. The user logs in, user agrees to scopes (what this token is allowed to access), and the user gets redirected back to the OAuth client with an authorization code in the URL. On the back end, the OAuth client sends the authorization code and client secret (we'll talk about client secrets shortly) to the authorization server, and the authorization server responds with the access token. That's the exact same flow, but using the new terminology we just discussed. Now let's talk specifics. We've seen what this flow looks like from the user's point of view, let's look at what it looks like from the developer's point of view. To use OAuth, you first need to register a new app. So for example, GitHub provides OAuth. If you want to create a new app for GitHub, you first register it. Different services require different types of data in the app registration, but every service will require at least an app name, because when the user goes to GitHub, for example, GitHub needs to be able to say "Amazon Web Services is requesting read access to your repos and gists" A redirect URI. And we'll talk about what that is shortly. GitHub will respond with: A client ID. This is a public ID that you'll be using to make requests A client secret. You'll be using this to authenticate your request. Awesome, you have registered your OAuth application. Let's say your app is YNAB, and one of your users wants to connect to Chase. So you start a new OAuth flow... your very first one! Step one: You will redirect them to Chase's authorization server's OAuth endpoint, passing these parameters in the URL: Client ID, which we just talked about. The redirect URI. Once the user is done on Chase, this is where Chase will redirect them back to. This will be a YNAB url, since you're the YNAB app. Response type, which is usually "code", because we usually want to get back an authorization code, not an access token, which is less secure. Scopes. So what scopes are we requesting? i.e. what user data do we want to access? This is enough information for the authorization server to validate the request and show the user a message like "YNAB is requesting read access to your accounts". How does the authorization server validate the request? Well, if the client ID isn't valid, the request is invalid right away. If the client ID is valid, the authorization server needs to check the redirect URI. Basically, since the client ID is public, anyone could go get the YNAB client ID, and create their own OAuth flow that hits Chase, but then returns the user back to, let's say, evildude.com. But that's why when you register your app, you have to tell Chase what a valid redirect URI looks like. At that point, you would tell Chase that only YNAB.com URIs are valid, thus preventing this evildude.com scenario. If everything is valid, the authorization server can use the client ID to get the app name, maybe the app icon, and then show a user consent screen. The user will click which accounts they want to give YNAB access to, and hit okay. Chase will redirect them back to the redirect URI that you gave, lets say ynab.com/oauth-callback?authorization_code=xyz. Side note: you might be wondering, what is the difference between URI and URL? Because I'm kind of using both. Well, a URL is any website URL that we know and love. URI is more general. URL is a type of URI, but there are many other types of URIs. The reason I'm saying redirect URI instead of redirect URL is because mobile apps won't have a URL. They'll just have a URI, which is a protocol they have made up that might look something like . So if you're only doing web work, whenever you read URI, you can read it as URL. And if you're doing mobile work, you can read URI and know that yes, your use case is supported too. So user is redirected back to ynab.com/oauth-callback?authorization_code=xyz, and now your app has an authorization code. You send that authorization code to the Chase authorization server, along with your client secret. Why include the client secret? Because again, the authorization code is in the URL. So anyone can see it and anyone could try to exchange it for the access token. That's why we need to send the client secret, so Chase's server can say "Oh yes I remember I had generated this code for this client ID, and the client secret matches. This is a valid request." And then it returns the access token. Note how in every step of the OAuth flow, they have thought through how someone could exploit the flow, and added safeguards*. That is a big reason why it's so complicated. *I'm reliably informed by a friend in security that the OAuth designers learned a bunch of lessons the hard way, and that is another reason why it is so complicated: because it had to be patched repeatedly. The other big reason is because we want the user to be involved. That makes it complicated because all the user stuff has to be frontend, which is insecure, because anyone can see it. And then all the secure stuff has to be on the back end. I keep saying frontend and back-end, but in the OAuth docs, they say front-channel and back-channel instead. Let's talk about why. Front-channel and back-channel So, OAuth doesn't use the terms frontend and back-end, it uses front-channel and back-channel. Front-channel means GET requests, where anyone can see the parameter in the URL, and back-channel means POST requests, where that data is encrypted (as part of the POST body). The reason OAuth doesn't use frontend or backend is, because you could make POST requests using JavaScript! So, theoretically, you could exchange your authorization code for an access token right on the frontend, in JavaScript, by making a POST fetch request. Now, there is a big problem with this, which is you also need the client secret to make that request. And of course, once the secret is on the frontend and accessible in JavaScript, it's not secret anymore. Anyone can access it. So, instead of using the client secret, there's a different way to do it called PKCE , spelled P-K-C-E, pronounced “pixie” (seriously). It's not as secure as doing it on the backend with the client secret, but if backend is not an option for you, you can do it using PKCE. So just know that if you have an app without a back-end, you can still do OAuth. I may cover PKCE in a future post, as it is now recommended for the standard flow as well, since it helps protect against auth code interception. Same problem for mobile apps. Unless you have a mobile app that has a backend component, like a backend server somewhere, if you're putting your client secret in a mobile app, well, anyone can get that because there are tons of tools to extract strings from mobile apps. So, instead of including your client secret in your app, you should again use PKCE to get that access token. So those are two other terms that are good to know: front-channel and back-channel . At this point, you've seen what the OAuth flow looks like from the user's perspective, and from the developer's perspective, and you have seen the components that make it secure. One last thing I want to mention is OAuth can look like a lot of different ways. I covered the main recommended OAuth flow above, but some people may do OAuth by passing back an access token in the redirect instead of the authorization token (doing that is called the "implicit flow"). Some people may do it using PKCE. There's even a way to do OAuth without the user consent part, but that really is not recommended. The other part of OAuth we didn't cover is that tokens expire and you need to refresh them. And that happens through a refresh flow. Also, OAuth is all about authorization, but some workflows use OAuth to log in, such as when you use a “sign-on with Google” feature. This uses OpenID Connect, or OIDC, which is a layer on top of OAuth that also returns user data instead of just an access token. I'm mentioning this here because when you look for OAuth on the web, you'll see a lot of different flows, and you may be confused as to why they're all different. And the reason is, OAuth is not straightforward like HTTP, OAuth can look a lot of different ways. Now you're good to go out and do your own OAuthing. Good luck! Thanks for reading DuckTyped! Subscribe for free to receive new posts and support my work. OAuth was first introduced in 2007. It was created at Twitter because Twitter wanted a way to allow third-party apps to post tweets on users' behalf. Take a second to imagine designing something like that today. How would you do it? One way would just be to ask the user for their username and password. So you create an unofficial Twitter client, and present the user a login screen that says "log in with Twitter". The user does so, but instead of logging into Twitter, they're actually sending their data to you, this third-party service which logs into Twitter for them. This is bad for a lot of reasons. Even if you trust a third-party app, what if they don't store your password correctly and someone steals it? You should never give your password to a third-party website like this. Another way you might be thinking is, what about API keys? Because you're hitting Twitter's API to post data for a user, and for an API , you use API keys . But API keys are general. What you need is an API key specific to a user. To solve these problems, OAuth was created. You'll see how it solves all these problems, but the crux of OAuth is an access token , which is sort of like an API key for a specific user. An app gets an access token, and then they can use that to take actions on the user's behalf, or access data for a user. How OAuth works OAuth can be used in a lot of different ways, one of the reasons it is so hard to understand. In this post, we’re going to look at a typical OAuth flow. The example I'm going to use is YNAB. If you haven't used it, YNAB is like a paid version of Mint. You connect it to a bank account, and then it pulls all your transactions from that account, and shows them to you with very pretty charts. You can categorize your spending, and then it tells you for example, hey, you're spending too much on groceries. It helps you manage your finances. So, I want to use YNAB, and I want to connect it to Chase Bank, but I don't want to give it my Chase password. So instead, I'm going to use OAuth. Let's look at the flow first, and then let's understand what's going on. We're actually going to look at the flow twice , because I think you need to look through an OAuth flow at least two times to understand what's going on. OAuth flow, take 1 So to start, I'm at YNAB, and I want to connect Chase as a source. The OAuth flow looks like this: YNAB redirects me to Chase. At Chase, I log in with my username and password. Chase shows me a screen saying "YNAB wants to connect to Chase. Pick what accounts you want to give YNAB access to". It'll show me a list of all my accounts. Let's say I pick just my checking account, to give YNAB read access to this account, and hit OK. From Chase, I'm redirected back to YNAB, and now, magically, YNAB is connected to Chase. This is the experience from a user's perspective. But what happened there? What magic happened in the background, so that YNAB somehow has access to my data on Chase? The end goal is to end up with an access token Remember, the end goal of OAuth is for YNAB to end up with an access token , so it can access my data from Chase. Somehow, as I went through this flow, YNAB ended up with an access token. I'll spoil the surprise by telling you how it got the access token, and then I'll walk you through what happened in more detail. A quick word on security How does Chase give YNAB the access token? When you were redirected from Chase back to YNAB, Chase could have just added the access token in the URL. It could have redirected you back to a URL like this: and then YNAB would be able to get the access token. BAD IDEA!! An access token is supposed to be secret, but URLs can end up in your browser's history or some server logs, in which case it's easy for anyone to see your access token. So Chase could technically redirect you back to YNAB with the access token in the URL , and then YNAB would have the access token. End of OAuth flow. But we don’t do it this way, because sending an access token in the URL is not secure. When you were redirected from Chase back to YNAB, Chase sent YNAB an authorization code in the URL. An authorization code is not an access token! Chase sends YNAB an authorization code, and YNAB exchanges the authorization code for an access token . It does this by making a backend request to Chase, a backend POST request over HTTPS, which means no one can see the access token. And then YNAB has the access token. End of OAuth flow. OAuth success. Two parts of OAuth Let's talk about what we just saw. At a high level, there are two parts to an OAuth flow. The first is the user consent flow , which is where you, the user, log in and pick what to give access to. This is a critical part of OAuth, because in OAuth, we always want the user to be actively involved and in control. The other part is the authorization code flow . This is the flow where YNAB actually gets this access token . Let's talk about more details of exactly how this works. And let's also talk about some terminology, because OAuth has very specific terminology. Instead of user, we say resource owner . Instead of app, we say OAuth client or OAuth app . The server where you log in is called the authorization server . The server where you get user data from is called the resource server (This could be the same as the authorization server). On the authorization server, when the user picks what's allowed, those are called scopes . I'll try to use that terminology, because you'll need to get familiar with it if you're going to read more OAuth documentation. So let’s look at this high level again, with the new terms. OAuth flow, take 2 You have OAuth clients. An OAuth client wants to access data on a resource server, and the data belongs to the resource owner. To do that, the OAuth client redirects to the authorization server. The user logs in, user agrees to scopes (what this token is allowed to access), and the user gets redirected back to the OAuth client with an authorization code in the URL. On the back end, the OAuth client sends the authorization code and client secret (we'll talk about client secrets shortly) to the authorization server, and the authorization server responds with the access token. That's the exact same flow, but using the new terminology we just discussed. Now let's talk specifics. We've seen what this flow looks like from the user's point of view, let's look at what it looks like from the developer's point of view. Registering a new app To use OAuth, you first need to register a new app. So for example, GitHub provides OAuth. If you want to create a new app for GitHub, you first register it. Different services require different types of data in the app registration, but every service will require at least an app name, because when the user goes to GitHub, for example, GitHub needs to be able to say "Amazon Web Services is requesting read access to your repos and gists" A redirect URI. And we'll talk about what that is shortly. A client ID. This is a public ID that you'll be using to make requests A client secret. You'll be using this to authenticate your request. Client ID, which we just talked about. The redirect URI. Once the user is done on Chase, this is where Chase will redirect them back to. This will be a YNAB url, since you're the YNAB app. Response type, which is usually "code", because we usually want to get back an authorization code, not an access token, which is less secure. Scopes. So what scopes are we requesting? i.e. what user data do we want to access?

Go Security API Tutorial JavaScript

▲ 0

0 views

DuckTyped 2 months ago

A colorful controversy

Here are three different images of blues. For each image, decide which blue you think is the warmer blue before moving on. Every Wednesday, I work with a local Minnesota artist who helps me improve my watercolor skills. One day, he was giving me some advice on my painting, and he said: "The sky needs to be a warm blue, so use ultramarine." This surprised me because I've always thought of ultramarine as a cool blue. So I conducted a poll on Instagram. I posted the same three images as above, and asked people to identify the warmer blue, just like I asked you. The results surprised me. They were almost evenly 50/50! Lets look at the results: Personally, I’d pick B, A, A. I was surprised at how split the crowd was. A couple of art friends messaged me, also surprised, and questioning whether they knew anything about art at all. Because as artists, it is so obvious what the warm blue is! Except as it turns out, it isn’t. Here’s the explanation: in each case, the blue on the right is the redder blue. I did a little Googling and found out this topic is hotly contested. Here's a thread specifically on ultramarine that goes on for 11 pages ! Lots of people saying they think Ultramarine is cool: I hear artists refer to Ultramarine as a warm blue and Phthalo as a cool blue. To me eyes, when I look at the two colors it seems just the opposite to me I believe Ultramarine is cool and Pthalo warm. This thread tries to bring some science into it: Those closer to yellow-orange/red-orange will be “warmer”, while those closer to blue-green will be “cooler”. This is why Ultramarine Blue (closer to red-orange) is warmer compared to Phathlo Blue (which is a blue-green). This is similar to what a few art friends told me: Red is a warm color, and green is a cool color, so blue + red = warmer, blue + green = cooler. Issue solved. But wait!! Here comes Reddit with a different opinion! It is a warm blue because it has yellow in it. Ah, so blue + purple = cooler, blue + yellow = warmer! Both make sense!! Which is right?? Well, you're going to hate this answer. Because not only is there not a right answer, the color wheel is wrong . Everyone knows the primaries, right? Red, yellow, and blue are primary colors, and they mix to form the secondary colors. Everyone knows that. Some color wheels that I painted. But if that's the case, why are printer cartridges filled with CMYK (Cyan, magenta, yellow, black)? Shouldn't they be RBYK? Here's the thing: the color wheel was created a long time ago, when artists first figured out they could mix colors to create new colors. At that time, they had limited pigments, and red, yellow, and blue were the best primary colors to use. We have many more pigments now. Red, yellow, and blue are no longer the best primaries: cyan, magenta, and yellow are. But we still keep teaching kids the wrong color wheel. And another thing: why do we have the same number of steps between yellow and blue as between yellow and red? I would say yellow to blue is a big shift (warm to cool), whereas yellow and red are much more similar. another color wheel I painted Now at this point, some of you are probably saying to yourselves, "I am so happy to be reading this. It's time that someone exposed the arrogance of the color wheel community." Others of you are probably saying to yourselves, "Wow, I've never read a blog post and been left with less information than I had before. When I started reading this, I didn't care about warm blues and I understood the color wheel. Now I don't understand either." For all of you in the second group, here is a possible solution to one problem. Some artists have been using the Munsell color wheel instead. This color wheel gives a lot more steps between yellow and blue. We actually have a lot of nuanced blue-green colors, like emerald, teal, etc. and it's useful to have a color wheel that gives space for those. Munsell wheel. Source: https://www.britannica.com/science/Munsell-color-system#/media/1/397642/148652 Maybe these extra steps will help being nuance to the warm blues discussion. Or maybe the reality is you cannot analyze color by itself, but only in relation to colors that is near , as some artists think. Maybe all blues can be warm or cool, depending on situation. We talked about color temperature. We talked about color wheels. Finally, here is a cool icebreaker for your next meeting: Show people the three images above, and ask them to pick the warmer blue. I guarantee you this will lead to an interesting discussion. Other interesting things: Colour story: Ultramarine My instagram The artist who’s teaching me watercolor Color fidget: a fidget that uses the Munsell wheel Thanks for reading DuckTyped! Subscribe for free to receive new posts and support my work. Story time Every Wednesday, I work with a local Minnesota artist who helps me improve my watercolor skills. One day, he was giving me some advice on my painting, and he said: "The sky needs to be a warm blue, so use ultramarine." This surprised me because I've always thought of ultramarine as a cool blue. So I conducted a poll on Instagram. I posted the same three images as above, and asked people to identify the warmer blue, just like I asked you. The results surprised me. They were almost evenly 50/50! Results Lets look at the results: Personally, I’d pick B, A, A. I was surprised at how split the crowd was. A couple of art friends messaged me, also surprised, and questioning whether they knew anything about art at all. Because as artists, it is so obvious what the warm blue is! Except as it turns out, it isn’t. Here’s the explanation: in each case, the blue on the right is the redder blue. Controversy!! I did a little Googling and found out this topic is hotly contested. Here's a thread specifically on ultramarine that goes on for 11 pages ! Lots of people saying they think Ultramarine is cool: I hear artists refer to Ultramarine as a warm blue and Phthalo as a cool blue. To me eyes, when I look at the two colors it seems just the opposite to me I believe Ultramarine is cool and Pthalo warm. This thread tries to bring some science into it: Those closer to yellow-orange/red-orange will be “warmer”, while those closer to blue-green will be “cooler”. This is why Ultramarine Blue (closer to red-orange) is warmer compared to Phathlo Blue (which is a blue-green). This is similar to what a few art friends told me: Red is a warm color, and green is a cool color, so blue + red = warmer, blue + green = cooler. Issue solved. But wait!! Here comes Reddit with a different opinion! It is a warm blue because it has yellow in it. Ah, so blue + purple = cooler, blue + yellow = warmer! Both make sense!! Which is right?? Well, you're going to hate this answer. Because not only is there not a right answer, the color wheel is wrong . The outdated color wheel Everyone knows the primaries, right? Red, yellow, and blue are primary colors, and they mix to form the secondary colors. Everyone knows that. Some color wheels that I painted. But if that's the case, why are printer cartridges filled with CMYK (Cyan, magenta, yellow, black)? Shouldn't they be RBYK? Here's the thing: the color wheel was created a long time ago, when artists first figured out they could mix colors to create new colors. At that time, they had limited pigments, and red, yellow, and blue were the best primary colors to use. We have many more pigments now. Red, yellow, and blue are no longer the best primaries: cyan, magenta, and yellow are. But we still keep teaching kids the wrong color wheel. And another thing: why do we have the same number of steps between yellow and blue as between yellow and red? I would say yellow to blue is a big shift (warm to cool), whereas yellow and red are much more similar. another color wheel I painted Now at this point, some of you are probably saying to yourselves, "I am so happy to be reading this. It's time that someone exposed the arrogance of the color wheel community." Others of you are probably saying to yourselves, "Wow, I've never read a blog post and been left with less information than I had before. When I started reading this, I didn't care about warm blues and I understood the color wheel. Now I don't understand either." For all of you in the second group, here is a possible solution to one problem. The Munsell color wheel Some artists have been using the Munsell color wheel instead. This color wheel gives a lot more steps between yellow and blue. We actually have a lot of nuanced blue-green colors, like emerald, teal, etc. and it's useful to have a color wheel that gives space for those. Munsell wheel. Source: https://www.britannica.com/science/Munsell-color-system#/media/1/397642/148652 Maybe these extra steps will help being nuance to the warm blues discussion. Or maybe the reality is you cannot analyze color by itself, but only in relation to colors that is near , as some artists think. Maybe all blues can be warm or cool, depending on situation. Final thoughts We talked about color temperature. We talked about color wheels. Finally, here is a cool icebreaker for your next meeting: Show people the three images above, and ask them to pick the warmer blue. I guarantee you this will lead to an interesting discussion. Other interesting things: Colour story: Ultramarine My instagram The artist who’s teaching me watercolor Color fidget: a fidget that uses the Munsell wheel

Design

▲ 0

0 views

DuckTyped 3 months ago

AWS in Terraform

Now that we have seen all the networking steps required, let's put it all together. But one last thing to mention first: One thing we haven't talked about is public IP addresses. All the IP addresses I've mentioned so far are local to within the VPC, but you also need some sort of public IP address that people on the internet can visit to connect to your server. You can create an EC2 instance with a public IP address. The key thing to note with public IPs though, is whenever you stop your instance, the public IP will change, so it’s not a good idea to point your domain name to your public IP . For that you need an Elastic IP . Request an Elastic IP from AWS, and then you can attach it to a particular EC2 instance (or to a load balancer, or a NAT gateway). I’ll show how to do this below. With that explained, here is the complete Terraform code you need to get a server up and running on AWS. You can get the entire thing in one file from here: ec2 with public IP ec2 with elastic IP Here's a visual summary of what we're going to do: EC2 with public IP example Boilerplate terraform initialization code: This is just boilerplate code you will need when using Terraform with AWS. VPC Now let’s create the VPC. It needs a CIDR block. The CIDR block can be anything, but I recommend using as the suffix. Subnet Create the subnet and associate it with the VPC. Notice the CIDR block for the subnet is a subset of the CIDR block for the VPC. Now we want to create the EC2 instance and put it in the subnet. Every EC2 instance needs an AMI (Amazon Machine Image). You can get the ID for the AMI you want from AWS here: https://console.aws.amazon.com/ec2/ Or you can just look it up in Terraform like this: EC2 instance Now let's create an EC2 instance that uses that AMI. Security group We'll also create a security group that allows inbound HTTP traffic on port 80 from anywhere: IGW Create an internet gateway and associate it with the VPC Route table Create a new route table and route and add a route to the internet gateway Associate our public subnet with this route table: Finally, we need to output the public IP of the instance so we can connect to it. Put all that in a file called . Run and . I'm glossing over the details of how to use Terraform here, since there are other tutorials on that. After the changes apply, Terraform will print out the IP address and URL. Hit the url that gets printed out using . You may need to give it a few minutes for the instance to boot up. If you get an error: If you get an error right away, that means everything works and you can hit your instance, but the server isn't up for some reason. If there's a wait before you get the error, that means you weren't able to connect to your instance at all. This could be any number of things, such as the IP you're using is wrong, or your security group or NACL are not set up to allow traffic in. All of the above, plus: Request an Elastic IP and associate it with your instance: And that's it! That is my introductory guide to networking for AWS. To close out, please enjoy this drawing of Nicholas Cage. Back to index Thanks for reading DuckTyped! Subscribe for free to receive new posts and support my work. ec2 with public IP ec2 with elastic IP EC2 with public IP example Boilerplate terraform initialization code: This is just boilerplate code you will need when using Terraform with AWS. VPC Now let’s create the VPC. It needs a CIDR block. The CIDR block can be anything, but I recommend using as the suffix. Subnet Create the subnet and associate it with the VPC. Notice the CIDR block for the subnet is a subset of the CIDR block for the VPC. AMI for EC2 instance Now we want to create the EC2 instance and put it in the subnet. Every EC2 instance needs an AMI (Amazon Machine Image). You can get the ID for the AMI you want from AWS here: https://console.aws.amazon.com/ec2/ Or you can just look it up in Terraform like this: EC2 instance Now let's create an EC2 instance that uses that AMI. Security group We'll also create a security group that allows inbound HTTP traffic on port 80 from anywhere: IGW Create an internet gateway and associate it with the VPC Route table Create a new route table and route and add a route to the internet gateway Associate our public subnet with this route table: Finally, we need to output the public IP of the instance so we can connect to it. Try it! Put all that in a file called . Run and . I'm glossing over the details of how to use Terraform here, since there are other tutorials on that. After the changes apply, Terraform will print out the IP address and URL. Hit the url that gets printed out using . You may need to give it a few minutes for the instance to boot up. If you get an error: If you get an error right away, that means everything works and you can hit your instance, but the server isn't up for some reason. If there's a wait before you get the error, that means you weren't able to connect to your instance at all. This could be any number of things, such as the IP you're using is wrong, or your security group or NACL are not set up to allow traffic in.

Cloud Terraform DevOps

▲ 0

0 views

DuckTyped 4 months ago

An illustrated guide to AWS Security Groups

This post is part of a series on AWS networking. Check out the other posts here . We’re connecting an EC2 instance to the internet. We have learned a ton, but we still can’t connect to the internet, because we haven’t taken care of… …security. Here’s where we are now: Setting up subnets, route tables etc is not enough. You also need to allow internet connections to pass through security: through your security groups, and through your NACLs. Those are two different things, and operate on different levels: Security groups operate at the firewall (or EC2 instance) level NACLs operate at the network (or subnet) level. Security Groups By default, your instance's security group is not set up for it to talk to the internet. You will have to explicitly set this up. To set up a security group to talk to the internet, you need to set up rules . A security group has inbound rules and outbound rules . Inbound requests are requests coming from outside to your EC2 instance. Outbound requests are requests going from your EC2 instance to the broader internet. Let’s look at an outbound rule that says you can connect to the internet. No inbound rules. That means that your servers can hit the internet, but no one from the internet can connect to your server. Here’s a security group that allows outbound traffic to anywhere: We’ll cover the options shortly. For now just know we just defined a security group with a rule using terraform, and that rules. Here’s another security group example. This one allows inbound traffic to port 80 from anywhere: You’ll want that one if you’re trying to put a server online. For the inbound rule above, we're saying that We allow connections only on port 80 The server is running on port 80 We allow the TCP protocol We allow connections from any IP. The outbound rule we had seen above is even more permissive. That rule allows connections from and to any port, using any protocol, from any IP. Note: On a toy server, you probably want that outbound rule so you can install packages on your EC2 instance. On production servers, people often don't have an outbound rule. This is so if a hacker gains access to the EC2 instance, they can't phone home. Users can still connect to their server because they have an inbound rule. Here’s one more useful one – ssh from anywhere: Useful so you can log on to your box. People often recommend making this more restrictive, so only your IP address is allowed to SSH onto the server. Going back to that outbound rule example. Now we can hit google.com from our server (an outbound request). But wait: the response from Google would be inbound traffic. And we have no inbound rules! nuance incoming Normally that inbound traffic wouldn’t be allowed, but since this is a response to your request, it is allowed. However, if Google wanted to initiate a connection, that would not be allowed. Sometimes you will hear security groups described as stateful . This is what that means. Stateful means the response to a request is allowed, even if inbound requests aren’t allowed. You don’t need to do anything special to enable this: it’s just how it works. You attach a security group to an instance. An instance can have multiple security groups attached to it, and you can attach a security group to multiple instances, so it's a many-to-many relationship. Now, before we move on to NACLs, here's a quick debugging tip. If you try to connect to your server and the connection just hangs for a long time, the issue is probably your security group. But if it connects but fails instantly, it's because your server is not running. We won't talk too much about Network ACLs, or NACLs, because the good news is, NACLs by default allow you to connect to the internet. You shouldn't need to do anything here. Most of the time, if you want to make a security change, you will be making it to your security group. So why use NACLs? It's useful to use a NACL when you want to block a specific IP. Notice that you can only allow connections using security groups. You can't create a rule that would deny a connection. That plus the fact that we use CIDR blocks to specify IP addresses, means you can't really block a single IP address using security groups. But you can using NACLs. With NACLs, you specify a bunch of rules with a priority level, and rules with a smaller priority level are applied first. You can use this functionality to block specific IPs. Summary You don't need to change your NACL. You do need to change your security group to explicitly allow inbound connections. You attach your security group to one or more EC2 instances. If you can't connect to your EC2 instance, and the request just hangs forever, it's probably because of your security group. Thanks for reading. In the next post, we'll put it all together, and finally talk to the internet! Chapter 6: Terraform Thanks for reading DuckTyped! Subscribe for free to receive new posts and support my work. Setting up subnets, route tables etc is not enough. You also need to allow internet connections to pass through security: through your security groups, and through your NACLs. Those are two different things, and operate on different levels: Security groups operate at the firewall (or EC2 instance) level NACLs operate at the network (or subnet) level. To set up a security group to talk to the internet, you need to set up rules . A security group has inbound rules and outbound rules . Inbound requests are requests coming from outside to your EC2 instance. Outbound requests are requests going from your EC2 instance to the broader internet. Here’s another security group example. This one allows inbound traffic to port 80 from anywhere: You’ll want that one if you’re trying to put a server online. Lets break down those fields For the inbound rule above, we're saying that We allow connections only on port 80 The server is running on port 80 We allow the TCP protocol We allow connections from any IP. Sometimes you will hear security groups described as stateful . This is what that means. Stateful means the response to a request is allowed, even if inbound requests aren’t allowed. You don’t need to do anything special to enable this: it’s just how it works. A couple misc notes You attach a security group to an instance. An instance can have multiple security groups attached to it, and you can attach a security group to multiple instances, so it's a many-to-many relationship. Now, before we move on to NACLs, here's a quick debugging tip. If you try to connect to your server and the connection just hangs for a long time, the issue is probably your security group. But if it connects but fails instantly, it's because your server is not running. Most of the time, if you want to make a security change, you will be making it to your security group. So why use NACLs? It's useful to use a NACL when you want to block a specific IP. Notice that you can only allow connections using security groups. You can't create a rule that would deny a connection. That plus the fact that we use CIDR blocks to specify IP addresses, means you can't really block a single IP address using security groups. But you can using NACLs. With NACLs, you specify a bunch of rules with a priority level, and rules with a smaller priority level are applied first. You can use this functionality to block specific IPs. Summary You don't need to change your NACL. You do need to change your security group to explicitly allow inbound connections. You attach your security group to one or more EC2 instances. If you can't connect to your EC2 instance, and the request just hangs forever, it's probably because of your security group.

DevOps Security Cloud

▲ 0

0 views

DuckTyped 6 months ago

An illustrated guide to route tables

Hi reader! This is part of a series on AWS Networking . The story so far: we've learned about VPCs . We've learned about subnets . We created a subnet and an internet gateway . Now we want to connect our subnet to the internet gateway. In order to do that, we need to learn about CIDR notation and route tables. This chapter is on route tables (“root tables”?). Make a new route table Add a rule to route to your internet gateway Associate this route table with a subnet Code for doing this Longer explanation: Someone wants to connect to your server. A packet comes in, with a destination IP address, and says "help I'm trying to go to 175.88.11.12". How do we know where the packet needs to go? Another packet comes up and says "help I'm trying to get to 142.250.191.110". This packet actually wants to go out to the internet. How do we direct this packet? The answer to both questions is a route table! A route table will map a destination to a target. Route tables are set up so given a destination, the route table tells you the next step to take . It's just like if you're trying to go to a city: In this example, the route table would look like this The destination you're trying to get to is Mumbai, and the next step you need to go to is the airport. Nuance! The route table doesn't tell the packet how to get to its final destination. It just tells the packet the NEXT STEP to take. We want to connect a subnet to an internet gateway. The route table for that might look like this: Lets break down what this table is saying. Suppose a packet comes in with a destination IP address. That IP address can be within your VPC, in which case the route table should route it locally The route table tells the packet where to go next. If it's any other IP address, it should route it to your internet gateway What does the route table for the logic above look like? Well, to start we need two routes: If our EC2 instance wants to hit an IP address inside our VPC, we route that request locally inside the VPC if it wants to hit any other IP address , we route that request to the internet, through the internet gateway Here's what the route table for the logic above would look like: You’ll notice we are using CIDR notation to specify a range of IP addresses (quick tutorial on CIDR here ). The first rule says, any IP address that starts with should get routed locally. The second rule says, all IP addresses should get routed to the internet gateway. You can add routes for specific IP addresses, but typically you will create routes for CIDR ranges. The route table in our example had two routes Notice that one of the routes matches any IP address while the other one matches only IP addresses in your VPC So if a packet comes in with a local IP address, technically it matches both routes. Where will it go? Route tables will always try to match the most specific route . Psst… how is the matching done? Using something called a Genmask. This isn't need-to-know information, so I won't describe it here, but now you know the word and can look it up if you want to. Route tables are created at the VPC level. This means VPCs can have many route tables. Every VPC comes with a route table, called the main route table . This route table will have a route by default that maps your VPC's CIDR range to the target. Typically, you want to leave this as is. All subnets are associated with the main route table by default. Suppose you have two subnets, both private. You're trying to make one of them public, i.e. giving it a connection to the internet. You do this by attaching it to an internet gateway. To do that , you need to add a route to your route table. Suppose you add a route like this to your main route table: This works, and one of your subnets is now public. Unfortunately, all of your subnets are now public! Since you made this change on the main route table , and all subnets are associated with the main route table by default, all your subnets are now public. To make a subnet public, you instead want to make a new route table and associate your subnet with that route table. But there's a gotcha here. Subnets can only be associated with one route table at a time , so now your subnet will no longer be associated with the main route table. Remember that the main route table had that route for local traffic. Now your public subnet can no longer handle local traffic! If you want the subnet to handle local traffic, you'll have to add a route to your new route table to handle local traffic . (Excuse the lack of syntax highlighting — Substack doesn’t have that) Here is the Terraform code for everything we have learned so far, including making a VPC, subnet, etc: Github gist If you are SSHed into an EC2 instance, you can see the route table by using the command. You can also do , which just calls in the background. Phew! You finally have a route in place, and your subnet is connected to the internet gateway. Here's what we have so far: Bullet points To connect a subnet to the internet, you need a route. Routes are defined as rules in a route table. A route table matches a destination to a target. Destinations are usually specified as CIDR ranges. Route tables are created on the VPC level. Each VPC comes with a default route table called the main route table. You probably want to create a new route table and associate it with a subnet. Subnets can only be associated with one route table at a time. Your subnet is now connected to the internet, but you still need to set up your security rules to allow traffic in. We’ll see that next! Chapter 5: Security groups Thanks for reading DuckTyped! Subscribe for free to receive new posts and support my work. Make a new route table Add a rule to route to your internet gateway Associate this route table with a subnet In this example, the route table would look like this The destination you're trying to get to is Mumbai, and the next step you need to go to is the airport. Nuance! The route table doesn't tell the packet how to get to its final destination. It just tells the packet the NEXT STEP to take. We want to connect a subnet to an internet gateway. The route table for that might look like this: Lets break down what this table is saying. Local or internet? Suppose a packet comes in with a destination IP address. That IP address can be within your VPC, in which case the route table should route it locally The route table tells the packet where to go next. If it's any other IP address, it should route it to your internet gateway What does the route table for the logic above look like? Well, to start we need two routes: If our EC2 instance wants to hit an IP address inside our VPC, we route that request locally inside the VPC if it wants to hit any other IP address , we route that request to the internet, through the internet gateway Here's what the route table for the logic above would look like: You’ll notice we are using CIDR notation to specify a range of IP addresses (quick tutorial on CIDR here ). The first rule says, any IP address that starts with should get routed locally. The second rule says, all IP addresses should get routed to the internet gateway. You can add routes for specific IP addresses, but typically you will create routes for CIDR ranges. Route tables pick the most specific route that matches The route table in our example had two routes Notice that one of the routes matches any IP address while the other one matches only IP addresses in your VPC So if a packet comes in with a local IP address, technically it matches both routes. Where will it go? Route tables will always try to match the most specific route . Psst… how is the matching done? Using something called a Genmask. This isn't need-to-know information, so I won't describe it here, but now you know the word and can look it up if you want to. The main route table Route tables are created at the VPC level. This means VPCs can have many route tables. Every VPC comes with a route table, called the main route table . This route table will have a route by default that maps your VPC's CIDR range to the target. Typically, you want to leave this as is. Subnets and route tables All subnets are associated with the main route table by default. Suppose you have two subnets, both private. You're trying to make one of them public, i.e. giving it a connection to the internet. You do this by attaching it to an internet gateway. To do that , you need to add a route to your route table. Suppose you add a route like this to your main route table: This works, and one of your subnets is now public. Unfortunately, all of your subnets are now public! Since you made this change on the main route table , and all subnets are associated with the main route table by default, all your subnets are now public. To make a subnet public, you instead want to make a new route table and associate your subnet with that route table. But there's a gotcha here. Subnets can only be associated with one route table at a time , so now your subnet will no longer be associated with the main route table. Remember that the main route table had that route for local traffic. Now your public subnet can no longer handle local traffic! If you want the subnet to handle local traffic, you'll have to add a route to your new route table to handle local traffic . Terraform (Excuse the lack of syntax highlighting — Substack doesn’t have that) Here is the Terraform code for everything we have learned so far, including making a VPC, subnet, etc: Github gist Bonus command line tip! If you are SSHed into an EC2 instance, you can see the route table by using the command. You can also do , which just calls in the background. Summary Phew! You finally have a route in place, and your subnet is connected to the internet gateway. Here's what we have so far: Bullet points To connect a subnet to the internet, you need a route. Routes are defined as rules in a route table. A route table matches a destination to a target. Destinations are usually specified as CIDR ranges. Route tables are created on the VPC level. Each VPC comes with a default route table called the main route table. You probably want to create a new route table and associate it with a subnet. Subnets can only be associated with one route table at a time.

Cloud Tutorial AWS

▲ 0

0 views

DuckTyped 6 months ago

The CIDR House Rules

Hello AWS adventurer, and welcome back to Chapter 3! So far we have learned about VPCs, internet gateways, and subnets. We have learned that a VPC is our own private network on AWS . We have learned that we need to create instances inside subnets. And finally, we have learned that to connect our servers to the internet, we need to connect our subnet to an internet gateway . In order to connect our subnet to an internet gateway, we need to use a route table. And to use a route table, we need to know about CIDR notation. So this chapter is about CIDR notation. What is CIDR notation? Well, to recap, this is an IP address Instead of a single address, CIDR notation lets you specify a range of IP addresses. This particular notation is equivalent to saying Let's dig into what this notation means. In an IP address, you have four numbers. Each number is 8 bits, so it can be a number from 0 to 255 Lets look at the CIDR range . The 16 means the first 16 bits are fixed , which means the last 16 bits can change. You can equivalently say this as the last two numbers can change, or replace the last two numbers with x's, or say this range All of those are the same. That is called a CIDR suffix . Here's another example This is a , so the first 30 bits are fixed, and the last two can change. Note that it's convention that the numbers that can change are written as zeros, but you don't need to. And, of course, when using CIDR notation, the last number can be a zero as well In that case, we're talking about all IP addresses. This is more commonly written like this You may see this notation in a route table. For example, if you have a route table where a connection to any IP address gets routed to an internet gateway, you would write it like that (We'll talk about route tables in the next chapter, so don't worry if you don't understand this image). BTW, each VPC you create will have a CIDR range. That means it's assigned a range of IP addresses, and the resources within that VPC have IP addresses that are somewhere in that range. For example, say the range for this VPC is to . Any resource in this VPC will have an IP address somewhere in that range. That means that every resource in this VPC will have an IP address that starts with . Subnets have their own CIDR ranges, and since each subnet is inside a VPC, its CIDR range is within the VPC's CIDR range. For example: Notice the is a smaller range than the , even though the number is bigger. That's all there is to it! Explanations for CIDR notation can get complicated, but at its heart, we just use it to specify a range of IP addresses. In the next chapter, we will learn more about route tables. We’re about halfway through our journey, at the end of which, we will be able to connect an EC2 instance to the internet. Yes, it’s a long journey. But I'm glad to be with you, Samwise Gamgee. CIDR notation lets you specify a range of IP addresses. Each number is 8 bits. The bigger the suffix, the smaller the range. Chapter 4: Route tables Thanks for reading DuckTyped! Subscribe for free to receive new posts and support my work. Hello AWS adventurer, and welcome back to Chapter 3! So far we have learned about VPCs, internet gateways, and subnets. We have learned that a VPC is our own private network on AWS . We have learned that we need to create instances inside subnets. And finally, we have learned that to connect our servers to the internet, we need to connect our subnet to an internet gateway . In order to connect our subnet to an internet gateway, we need to use a route table. And to use a route table, we need to know about CIDR notation. So this chapter is about CIDR notation. What is CIDR notation? Well, to recap, this is an IP address Instead of a single address, CIDR notation lets you specify a range of IP addresses. This particular notation is equivalent to saying Let's dig into what this notation means. In an IP address, you have four numbers. Each number is 8 bits, so it can be a number from 0 to 255 Lets look at the CIDR range . The 16 means the first 16 bits are fixed , which means the last 16 bits can change. You can equivalently say this as the last two numbers can change, or replace the last two numbers with x's, or say this range All of those are the same. That is called a CIDR suffix . Here's another example This is a , so the first 30 bits are fixed, and the last two can change. Note that it's convention that the numbers that can change are written as zeros, but you don't need to. And, of course, when using CIDR notation, the last number can be a zero as well In that case, we're talking about all IP addresses. This is more commonly written like this You may see this notation in a route table. For example, if you have a route table where a connection to any IP address gets routed to an internet gateway, you would write it like that (We'll talk about route tables in the next chapter, so don't worry if you don't understand this image). VPC CIDR Ranges BTW, each VPC you create will have a CIDR range. That means it's assigned a range of IP addresses, and the resources within that VPC have IP addresses that are somewhere in that range. For example, say the range for this VPC is to . Any resource in this VPC will have an IP address somewhere in that range. That means that every resource in this VPC will have an IP address that starts with . Subnets have their own CIDR ranges, and since each subnet is inside a VPC, its CIDR range is within the VPC's CIDR range. For example: Notice the is a smaller range than the , even though the number is bigger. That's all there is to it! Explanations for CIDR notation can get complicated, but at its heart, we just use it to specify a range of IP addresses. In the next chapter, we will learn more about route tables. We’re about halfway through our journey, at the end of which, we will be able to connect an EC2 instance to the internet. Yes, it’s a long journey. But I'm glad to be with you, Samwise Gamgee. Summary CIDR notation lets you specify a range of IP addresses. Each number is 8 bits. The bigger the suffix, the smaller the range.

Cloud Tutorial

▲ 0

0 views

DuckTyped 7 months ago

Old man yells at subnets

Back to index In the last chapter, we talked about VPCs, and how a VPC is a private network inside AWS. It's like a big bubble of protection that we've built around our instances. There are other people using AWS, and some of them might be bad people who want to connect to our instances and steal our data. But our instances are in a private network that no one else can get into. If you haven't read the chapter on VPCs, Read the chapter on VPCs The benefit of having a protective bubble is that no one can connect to our instances. The downside of having a protective bubble is that no one can connect to our instances! We're building a website, after all, aren't we? How's anyone supposed to connect to our servers with this darned bubble in place? We need to poke a hole in our bubble so we can talk to the internet. An internet gateway is like a hole you poke in your VPC so you can talk to the internet. End of section. Terraform code: Now you have an internet gateway (aka an IGW, pronounced "igwoo"). Here's what we have so far: Here's what the internet gateway looks like in the AWS docs: Let's zoom in: Zoom in some more: Beautiful. Like the Gateway of India , but online. So now we have an internet gateway (IGW), we can connect to the internet, right? No! Not even close! Muahahahaaaaaa Let's zoom out. First we need instances. But before that (so zeroth), we need a place to put them in. See that green box labeled subnet ? You can't connect your EC2 instance to an internet gateway directly! Only a subnet can be connected to the internet gateway. You need to put your instance inside a subnet, and connect that subnet to the internet gateway. A subnet is a way to group your instances. It's sort of like tagging them. Here I have four instances and now I've tagged them with subnet A or subnet B To connect an instance to the internet, you need to put it in a subnet that is connected to the internet. Notice I said "put it in a subnet", but the subnet's not like a basket you're putting your instances into. It's just a tag, or a connection. Or an assignment. By default, subnets can't talk to the internet. Subnets without a connection to the internet are called private subnets . Subnets that have a connection to the internet are called public subnets . To get your instance connected to the internet, one of the steps you need to take is assign it to a public subnet. Sidebar: Why do we need to put our instance in a subnet? Well, we can't just put an instance in a VPC, because remember, the VPC spans all the availability zones in a region. To create an instance, we need to specify which availability zone we want to create the instance in. That way, we can create instances in multiple availability zones, to guard against outages. VPCs are region-scoped, but subnets are availability zone-scoped . To assign our instance an AZ, we assign it a subnet. Now we have something like this And to connect to the internet here's what we need: Everything is ready, but to connect the subnet to the internet gateway, we need a route. Terraform code for creating a subnet: A route is a connection from your subnet to the internet gateway so that your subnet can talk to the internet. We need to create a route from our subnet to the internet gateway. To do that, we need to learn two new concepts first: CIDR notation , and route tables . The next chapter will talk about these stimulating concepts. An internet gateway is like a hole you poke in your VPC so you can talk to the internet. You can't connect your EC2 instance to an internet gateway directly. You need to put it in a subnet, and connect that subnet to the IGW. Subnets without a connection to the internet are called private subnets . Subnets with a connection to the internet are called public subnets . Chapter 3: CIDR Thanks for reading DuckTyped! Subscribe for free to receive new posts and support my work. Here's what the internet gateway looks like in the AWS docs: Let's zoom in: Zoom in some more: Beautiful. Like the Gateway of India , but online. So now we have an internet gateway (IGW), we can connect to the internet, right? No! Not even close! Muahahahaaaaaa Let's zoom out. First we need instances. But before that (so zeroth), we need a place to put them in. See that green box labeled subnet ? Subnets You can't connect your EC2 instance to an internet gateway directly! Only a subnet can be connected to the internet gateway. You need to put your instance inside a subnet, and connect that subnet to the internet gateway. A subnet is a way to group your instances. It's sort of like tagging them. Here I have four instances and now I've tagged them with subnet A or subnet B To connect an instance to the internet, you need to put it in a subnet that is connected to the internet. Notice I said "put it in a subnet", but the subnet's not like a basket you're putting your instances into. It's just a tag, or a connection. Or an assignment. Public and private subnets By default, subnets can't talk to the internet. Subnets without a connection to the internet are called private subnets . Subnets that have a connection to the internet are called public subnets . To get your instance connected to the internet, one of the steps you need to take is assign it to a public subnet. Sidebar: Why do we need to put our instance in a subnet? Well, we can't just put an instance in a VPC, because remember, the VPC spans all the availability zones in a region. To create an instance, we need to specify which availability zone we want to create the instance in. That way, we can create instances in multiple availability zones, to guard against outages. VPCs are region-scoped, but subnets are availability zone-scoped . To assign our instance an AZ, we assign it a subnet. Now we have something like this And to connect to the internet here's what we need: Everything is ready, but to connect the subnet to the internet gateway, we need a route. Terraform code for creating a subnet: Routes A route is a connection from your subnet to the internet gateway so that your subnet can talk to the internet. We need to create a route from our subnet to the internet gateway. To do that, we need to learn two new concepts first: CIDR notation , and route tables . The next chapter will talk about these stimulating concepts. Summary An internet gateway is like a hole you poke in your VPC so you can talk to the internet. You can't connect your EC2 instance to an internet gateway directly. You need to put it in a subnet, and connect that subnet to the IGW. Subnets without a connection to the internet are called private subnets . Subnets with a connection to the internet are called public subnets .

DevOps Cloud

▲ 0

0 views

DuckTyped 8 months ago

An illustrated guide to Amazon VPCs: bonus content

This post contains some bonus info for An illustrated guide to Amazon VPCs . One reader added some fascinating insights: Security groups were the original solution to the second problem (that everyone's servers were in the same network). The address conflict problem transformed into another problem after VPCs. After everybody got default VPCs, companies went from conflicting with each other to conflicting with themselves. Now they have a bunch of instances with the same IPs, in different VPCs, that still need to talk to each other… a problem that needed double-sided NAT. A big part of the first problem (IP address conflicts) was actually that AWS was about to run out of IP addresses to issue to customers. Back then, every instance got a public address because there was no other way to ssh into your instance. Because public addresses are so expensive and limited, this was a scaling cliff for AWS, and passed on as an unnecessary cost to customers. VPCs let you use a single bastion with a public IP to access the rest of your hosts.

AWS Cloud

▲ 0

0 views

DuckTyped 9 months ago

Old man yells at cloud: a mini-book on AWS networking

If you have ever tried to put an app on AWS, you know that you first need to absorb a ton of knowledge about AWS networking. Before you set up that first EC2 instance, you need to know terms like VPCs, subnets, security groups, and internet gateways. You need to remember how CIDR notation works, and that networking class you took suddenly feels like a long time ago. Without a solid understanding of AWS networking, you may be able to start up an EC2 instance, but you will have a hard time getting it connected to the internet. This series of posts will get you up to speed on AWS networking, so you can get an app up on AWS. Every post is meant to be easy to understand, and spends time really explaining the concepts, so you understand how things work, and can adapt them to fit your use case. Each post is full of illustrations that I hope will brighten your journey as you learn AWS networking. Good luck and happy reading! Read Chapter 1: VPCs Route tables Security groups AWS in Terraform Thanks for reading DuckTyped! Subscribe for free to receive new posts and support my work. Route tables Security groups AWS in Terraform

AWS Tutorial Cloud

▲ 0

0 views

DuckTyped 9 months ago

An illustrated guide to Amazon VPCs

Back to index In this section, I talk about why VPCs were invented and how they work. This is critical to understand because almost everything you do in AWS will happen inside of VPC. If you don't understand VPCs, it will be difficult to understand any of the other networking concepts. If you're reading this, maybe you have one of these and you just found out that to put your app on AWS, you need all of this: And you have no idea what VPCs, subnets and so on are. I'll help you learn about all those pieces. A little about me, I’m a long-tailed duck, and I run a business selling phones to hackers, called Blackhatberry . Now let's get started. This is the story of VPCs (Virtual Private Cloud)s, our first big topic. Many moons (and suns) ago, some AWS engineers were sitting in a room. They had a serious issue. "Guys, lets talk business. Why aren't more companies moving to AWS?" they said. "Maybe because all instances run in a single shared network, which means users can access each other's instances, and see each other's data," someone said. "Maybe because it's hard for them to move their existing servers to AWS, because of IP address conflicts," someone else said. "Wait… what are IP address conflicts?” “And existing servers? Shouldn’t they be moving to our servers?” This is the first reason people weren’t switching to AWS. Here's what I mean by IP address conflicts. I own a bunch of servers for Blackhatberry. One of them has the IP address `172.98.0.1`. Now, my neighbor also has a server for her business. She loves my ip address. “Ah, 172.98.0.1, what a beautiful destination,” she says. So she copies my address! Now we both have servers with the same IP address! Sidebar: You can find your local IP address using `ipconfig getifaddr en1` (works for Macs for wireless internet connections). Now you're thinking "so what?". And actually... you're totally right. Even though our servers have the same IP address, they are in different networks, so it's not an issue. But here comes trouble . Because we both want to get on AWS. But if we have two servers with the same IP address on AWS, that's a problem! Bam: IP address conflict. Every server in a network needs to have a unique IP address for the same reason that every house in a city needs to have a unique address. Otherwise, if someone has a package, they wouldn't know which house to deliver it to. Now I know what you're thinking. Why don't you just get new servers in AWS? Why connect your on-prem ("on premises") servers to AWS? Isn't the whole point that you're moving to AWS? And sure, we can afford to do that, but there are companies with dozens of on-prem servers. Migrating everything to AWS is simply not an option for them. They need functionality so they can create new servers in AWS, but also connect their existing , on-prem servers into the same network . And this does not work when everyone is part of the same network, because of IP address conflicts. This was a huge problem for AWS! I mean, see how serious these engineers look: This IP conflict issue meant people with on-prem servers had no easy way to gradually move to AWS. Think of all the potential customers they were losing! I'm giving you this background so you can understand why VPCs were invented. IP address conflicts weren't the only issue. In AWS, everyone's servers used to be on the same network, which meant if you were careless, it was easy for anyone to connect to your server and look at all kinds of sensitive data! For both these reasons, Amazon needed to give each customer their own private network, instead of having them all on the same shared network. And so VPCs were born. So there are two problems we're trying to solve: IP address conflicts The fact that users can access each other's instances because they're in one big shared network. Remember: duplicate IPs were totally fine when my network and my neighbor's network were separate. What Amazon needed was a way to give each person their own private network, but inside AWS . That way, they could bring their IP addresses with them, and they wouldn't conflict with anyone else's IP addresses. Maybe you're wondering, "why can't we just change the IP addresses so all machines have a unique IP address?" Well, in networking, you set up some things based around specific IP addresses (I'll get to exactly what stuff later), so that idea would require a lot of work in practice. Separate networks would also solve the security problem. By the way, why am I spending so long on VPCs? Isn't this post about putting one of these on one of these? Two reasons. Because everything we will build happens in a VPC, so it's the starting point for things. Because a VPC is not something you can see, and I like to visualize my internet architecture. Other people visualize it in a way that's really confusing for me, and I want to make it less confusing for you. I have read guides (such as the AWS docs ) where people visualize a VPC like this. First, they'll say, "Oh yeah, I created a new VPC with four subnets inside it, in two availability zones". And they'll draw an image that looks like this: But less pretty obviously – this is what theirs look like: (Taken from the AWS VPC docs ) a region is a place you can go to, and an availability zone has data centers you can walk inside, that hold lots of servers. Both of those are physical places. But what is the VPC? Is it a big tarp that sits on top of the data centers? Is it a dark fog? Is it a general feeling of unease that blankets the region, as all the data centers play Radiohead's "Fitter, Happier" on repeat? …what is it? We've talked about why AWS needed VPCs, and the idea behind VPCs, but how are they implemented? How do they actually work? Your instances in AWS always run inside a VPC. But in real life, of course, your instances are just running on servers in AWS datacenters. You may have many instances running on many different servers. How does Amazon connect these instances, across different servers , into their own private network? It uses something called the mapping service . Suppose I have an instance A on server 1, and I want to talk to another instance B on server 2. All instance A knows is the IP address for instance B. It hits the mapping service with that IP address. The mapping service then checks what VPC instance A is in, finds the instance with that IP in that VPC , and forwards the request to it. That "in that VPC" part is important. The mapping service makes it so me and my neighbor can both have instances on AWS with the same IP, but when I hit that IP address, the mapping service will connect me to the instance in my VPC , and when my neighbor hits that IP address, the mapping service connects them to the instance in their VPC . The mapping service is what ensures that we can never connect to each other's instances. Through the mapping service, all my instances are connected together, and they can have any IP address I want, because it's like they're namespaced to me. The mapping service is what creates the private network inside AWS for me. So when you think VPC, picture a service that connects all these instances together. Going back to this image, we can now understand what it means: That VPC box just means the scope of the mapping service. A mapping service can connect EC2 instances that are on servers in different availability zones, which is why the VPC is overlaid over the two availability zones. But the mapping service can't connect instances in different regions, so the VPC doesn't span regions. Today everything happens in VPCs. Your instances are always in a VPC. Everyone gets a default VPC when they open an AWS account. We no longer have the issue where users can access each other's instances, and we don't have the IP collision issue either. So we find out that they don't play "Fitter, Happier" in the data centers after all. Maybe it's just a recording of Jeff Bezos singing "Money" by Pink Floyd. Terraform code Throughout this guide, I'll show you how to create AWS resources using Terraform. I find Terraform easier to follow than point-and-click on the AWS console, because you can just copy the code and run it. Here's the Terraform code to create a VPC: In any Terraform file, you'll also need a couple of boilerplate blocks for and . The full code listing is here . You can apply this code to create a new VPC. Ignore the part for now, I'll discuss CIDR in more detail in a future section. In AWS, every customer has their own private network called the VPC. Without private networks, we run into IP address collisions. Without private networks, everyone is on the same network, which is really bad for security. VPCs are implemented using the mapping service. Chapter 2: subnets P.S. A reader provided me with some extra context on AWS VPCs. You can read that here . Thanks for reading DuckTyped! Subscribe for free to receive new posts and support my work. And you have no idea what VPCs, subnets and so on are. I'll help you learn about all those pieces. A little about me, I’m a long-tailed duck, and I run a business selling phones to hackers, called Blackhatberry . Now let's get started. This is the story of VPCs (Virtual Private Cloud)s, our first big topic. Many moons (and suns) ago, some AWS engineers were sitting in a room. They had a serious issue. "Guys, lets talk business. Why aren't more companies moving to AWS?" they said. "Maybe because all instances run in a single shared network, which means users can access each other's instances, and see each other's data," someone said. "Maybe because it's hard for them to move their existing servers to AWS, because of IP address conflicts," someone else said. "Wait… what are IP address conflicts?” “And existing servers? Shouldn’t they be moving to our servers?” IP address conflicts This is the first reason people weren’t switching to AWS. Here's what I mean by IP address conflicts. I own a bunch of servers for Blackhatberry. One of them has the IP address `172.98.0.1`. Now, my neighbor also has a server for her business. She loves my ip address. “Ah, 172.98.0.1, what a beautiful destination,” she says. So she copies my address! Now we both have servers with the same IP address! Sidebar: You can find your local IP address using `ipconfig getifaddr en1` (works for Macs for wireless internet connections). Now you're thinking "so what?". And actually... you're totally right. Even though our servers have the same IP address, they are in different networks, so it's not an issue. But here comes trouble . Because we both want to get on AWS. But if we have two servers with the same IP address on AWS, that's a problem! Bam: IP address conflict. Every server in a network needs to have a unique IP address for the same reason that every house in a city needs to have a unique address. Otherwise, if someone has a package, they wouldn't know which house to deliver it to. Now I know what you're thinking. Why don't you just get new servers in AWS? Why connect your on-prem ("on premises") servers to AWS? Isn't the whole point that you're moving to AWS? And sure, we can afford to do that, but there are companies with dozens of on-prem servers. Migrating everything to AWS is simply not an option for them. They need functionality so they can create new servers in AWS, but also connect their existing , on-prem servers into the same network . And this does not work when everyone is part of the same network, because of IP address conflicts. This was a huge problem for AWS! I mean, see how serious these engineers look: This IP conflict issue meant people with on-prem servers had no easy way to gradually move to AWS. Think of all the potential customers they were losing! I'm giving you this background so you can understand why VPCs were invented. IP address conflicts weren't the only issue. In AWS, everyone's servers used to be on the same network, which meant if you were careless, it was easy for anyone to connect to your server and look at all kinds of sensitive data! For both these reasons, Amazon needed to give each customer their own private network, instead of having them all on the same shared network. And so VPCs were born. Why do they call it a VPC if you can't see it! So there are two problems we're trying to solve: IP address conflicts The fact that users can access each other's instances because they're in one big shared network. Because everything we will build happens in a VPC, so it's the starting point for things. Because a VPC is not something you can see, and I like to visualize my internet architecture. Other people visualize it in a way that's really confusing for me, and I want to make it less confusing for you. But less pretty obviously – this is what theirs look like: (Taken from the AWS VPC docs ) Now, a region is a place you can go to, and an availability zone has data centers you can walk inside, that hold lots of servers. You may have many instances running on many different servers. How does Amazon connect these instances, across different servers , into their own private network? It uses something called the mapping service . Mapping service Suppose I have an instance A on server 1, and I want to talk to another instance B on server 2. All instance A knows is the IP address for instance B. It hits the mapping service with that IP address. The mapping service then checks what VPC instance A is in, finds the instance with that IP in that VPC , and forwards the request to it. That "in that VPC" part is important. The mapping service makes it so me and my neighbor can both have instances on AWS with the same IP, but when I hit that IP address, the mapping service will connect me to the instance in my VPC , and when my neighbor hits that IP address, the mapping service connects them to the instance in their VPC . The mapping service is what isolates our servers. The mapping service is what ensures that we can never connect to each other's instances. Through the mapping service, all my instances are connected together, and they can have any IP address I want, because it's like they're namespaced to me. The mapping service is what creates the private network inside AWS for me. So when you think VPC, picture a service that connects all these instances together. Going back to this image, we can now understand what it means: That VPC box just means the scope of the mapping service. A mapping service can connect EC2 instances that are on servers in different availability zones, which is why the VPC is overlaid over the two availability zones. But the mapping service can't connect instances in different regions, so the VPC doesn't span regions. Today everything happens in VPCs. Your instances are always in a VPC. Everyone gets a default VPC when they open an AWS account. We no longer have the issue where users can access each other's instances, and we don't have the IP collision issue either. So we find out that they don't play "Fitter, Happier" in the data centers after all. Maybe it's just a recording of Jeff Bezos singing "Money" by Pink Floyd. Terraform code Throughout this guide, I'll show you how to create AWS resources using Terraform. I find Terraform easier to follow than point-and-click on the AWS console, because you can just copy the code and run it. Here's the Terraform code to create a VPC: In any Terraform file, you'll also need a couple of boilerplate blocks for and . The full code listing is here . You can apply this code to create a new VPC. Ignore the part for now, I'll discuss CIDR in more detail in a future section. Summary In AWS, every customer has their own private network called the VPC. Without private networks, we run into IP address collisions. Without private networks, everyone is on the same network, which is really bad for security. VPCs are implemented using the mapping service.

Tutorial Cloud

▲ 0

0 views