Go For Real World Applications

1. Intro: Go is an amazingly powerful back end language that can be an asset to any developer's tool kit. It's performant. It has built in serialization and deserialization. JSON, it has testing built in and much more. In this course, we're going to start completely from scratch, making a new folder, scaffolding a G project, writing all the code for it, writing tests for the project. Dockerizing the application, and then finally in the end, automating a testing and building workflow CICD pipeline with CircleCI. By the end of the course, I hope you'll gain a practical and hands on overview of how you can create dockerized go applications, including tests. I hope you'll join me in this course, Go for real world applications. 2. Why Go: Before we even get started talking about what our application is going to do or any programming, I want to quickly go over why we would even choose G as a language choice in the first place. So why would we choose G? First off, it's extremely performant. At this point, I've written at least a dozen go applications, and never once have I needed to hunt down any sort of race condition or performance issue. And one of those was an API for a fairly large startup. We had about 10,000 users, and at some points, 1,000 to 2000 simultaneous users. And even then, we never had any major performance issues, and that was running on a four core virtual machine. Go is also extremely compact. So the application runs from typically a single main dot go file, and that can be done simply by issuing go run main dot go. As we'll see later on in the course, this is extremely powerful for when we want to dockerize our app. It's very simple to transport and run a go application. Go is also extremely testable. It has a testing framework built in. So we would import this testing package. We pass that in to our testing function. And then it's as simple as calling the function that we want to test. And in this case, I've returned an error from this function. If the error is not Null, this is go for not null or empty. Then we use this pointer to the testing framework, and we issue an error. So this is built in. We don't need any external config. We don't need any external libraries. It is built into the language. Go is also extremely API friendly. So I have an example here to show you how it works in Go. So if we assume we have some sort of JSON response from an API like this, we can use the JSON package and also the JSON identifier when we define a type here, call it some API, it's a structure. And we know that our response has some type int and some array of string. It's a string array. We use this JSON identifier. Now, the magic happens when we use these identifiers and we use the built in JSON package, let's assume that we already have the response from the API, some sort of binary result. We can simply call JSON Unmarshal and Golang will serialize that binary result into our response object. And when it sees these keys, it knows to put those values in each of these parts of this struct. So then we went to assuming no error, marshalling the response, if we went on to log out some int, we would get five and if we looped over the array of strings, we would get this A, B, and C. So again, we don't need any config files, any external libraries. We just use the built in JSON library with the Unmarshal method, and Golang handles all the serialization and deserialization of JSON right out of the box. Going is also extremely uniform. So when you're writing go with a modern code editor like Visual Studio Code, go formats on save according to its own built in rules. And this means that no matter where you're reading G code, whether it's an external package, a third party package, your teammates code, you will always see the same patterns, indentation and spacing on any G code. This makes it very easy to read other go projects. This also saves you and your team time from debating about formatting rules like tabs and spaces and all these things. These rules are also built into the go language. So in summary, G is very performant. It's very fast, at least in all the experience I've used it, it's very compact. You just need your single main dot go file. It's easily testable. Testing comes with the language itself. It's very API friendly. This JSON serialization and deserialization also comes right out of the box with the language, and it's uniform. These linting and formatting rules also come with the language. And so altogether, G is a very solid choice for any back end software that you may need to build. So with a basic background of what G is and some of its advantages, in the next lesson, we can start looking at the actual application that we'll build, and then we can finally get into coding the application. 3. Allergy API: This course, we're going to build an application that messages us and warns us daily of the expected pollen levels in the air. If you're like me, you may suffer from hay fever from about mid May to mid June each year, at least in the northern hemisphere. And don't worry. Even if you have no allergies whatsoever, I promise that the content and patterns in this course are extremely useful no matter what type of go application you're using. So where I live in Austria, the Medicine University of Vienna has this great site that predicts the pollen levels for the day. And with some digging and inspection of the API calls for this site, I found the endpoints that produce all the data. There are mainly two endpoints that we'll be using. The first is this get hourly load data, which has a success key and the result. Most importantly, here is this hourly array. So for each hour, it'll give an expected value of the pollen in the air, ranging from zero to, I believe, eight or nine. The second endpoint that we'll be utilizing in this course is this get current chart data. This one we'll be using for historical averages. So as you can see here, we also have a success key. We have some results. And for each result, we have a date, the current or in this case, what actually happened on that date this year, and then the historical average. However, it's May 16 right now, and as you can see, the current is zero, and even for yesterday, the current is zero. So this data is a little bit delayed. However, as I mentioned, we'll be using it mainly for the average values. So we'll be messaging out both the expected value for the day from this hourly data and also the historical average for the day. So with an overview and an idea of what our app is going to be in the next lesson, we'll briefly discuss the technical requirements that we want our app to fulfill, and then we can finally start getting into writing some code. 4. Application Requirements: The last lesson, we looked at the allergy API we were going to use and where that data was coming from. And in this lesson, we'll just briefly discuss the application requirements at a more technical level before we get starting to code. So from an application flow standpoint, at a specified daily time, we want to call those allergy endpoints. We're going to parse that JSON response from each call and then do some formatting work to transform it into a nice human readable message. Then finally, we want to send that nice message via Slack webhooks. All the functions that we write should have tests. The application should be run from a docker container. The docker container should be restarted on any sort of failure or crash. An automated pipeline should be built that runs all the tests we wrote and then publishes the Docker container in the case that all those tests indeed pass. And then finally, the various important values used around the application, such as the API endpoints, the Slack web hook URL, the Cron Job time and Crown job time zone should all be configurable. So throughout this course, we will step by step tackle and meet all of these requirements. In the next lesson, we will finally get started scaffolding our go application. 5. Install Go and Visual Studio Go Extension: Get started, we should all check that we have G installed. To check that I have G installed, I'll just simply issue G version. And I can see I have G 1.20 0.2. If this command doesn't return anything or an error, then you probably need to install G on your system. You can go to go dot dev slash DClash Install, and they have the installation instructions for Linux, Mac and Windows. For the remainder of this course, I'll assume that you're coding along with me in Visual Studio code. So I also recommend that you have the G extension installed. To find the G extension, head over to the extensions tab and search for G. It should be the first result, and of course, I have it installed already. This extension is a one stop shop for everything that you should need for editing G code. It's a combined inter, tester, debugger, and format for G. Once you've installed this extension, we can begin scaffolding our G project. 6. Scaffolding the Project: The last lesson, we made sure that we had G installed, and we also installed the Visual Studio Code G extension. Now we will scaffold our app. Typically, when you start a new G project, you would create a new folder for that application. In this case, I would call it Allergy Cron and you would issue make DR Allergy cron. However, since I'm already in the course repository, I have a folder here for the project and I won't make a new folder. The next step would then be to initialize a module for the project. I typically take the same name as the folder name itself. In your case, that would be allergy cron. So here, we'll initialize it according to that folder name. The command to initialize a G module is Gomd init and then in our case, Allergy Crown. Go will tell us that it created a new go dot mod file. We can hop in this gomd file and see what's in it. For now, since we haven't installed any additional packages, it just has the module name and the go version that was used to create it. Next, we can create the main file, so that is touch main dot go. And if you were still in just a terminal environment, you didn't have an editor open. You could, of course, open the main dot go with the Open command. Or if you wanted to open it in Visual Studio code, you could issue code main dot go. So after all that setup, we are finally ready to actually start writing some G code. 7. Building the Cron Job: The last lesson, we scaffolded our new G project, initializing a G module, which left us with this go dot mod file, and we created an empty main dot go file. In this lesson, we'll start tackling the first technical requirement, and that is to have a daily cron job that fires exactly at the same time. Every go file has a package declaration at the very top. In this case, this is package Min. Then you might have some imports here. And in the case of the main dot go file, we'll have our main function, which is just simply function main. Now, for Crown jobs and G, I like to use the popular Rob Fig Cron library. This library also enables us to set the Cron Job according to a specific time zone. So first, I'll make use of Go's time package to load the time zone, and then we can set up the crown job in that time zone. So we get a location and an error and that's from time, load location. And since I'm in Austria, I want Europe, Vienna. Then, of course, we need to check if the error is not Nil, then we're going to panic with that error, and otherwise, we can continue on. Now, to set up a Crown Job with this package, all we need to do is call new with location, and we're going to pass in that location. Now, we see with the inter, as soon as I make reference to this cron, the inter knows which package that I want to use. The one issue here is that this package isn't in our mod file yet. So if we hover over the import here, Go complains that there's no module providing this package. But, luckily, the Linter provides us with a quick fix, which is simply the install or the go get package command. So if we just click that the package is installed. You should see your got mod file modified, and then we can continue on with our code here. Not to pass the actual Cron timing, we need to add Cron Job dot ad Funk and this takes the standard unix type cron identifier with six characters. And those are specifically the seconds, the minutes, the hours, the day of month, the month itself, and the day of week. In our case, I think a reasonable time, at least for me is 8:00 A.M. Every morning. So I will specify both zero for the seconds and minutes, and then eight for the hour. And then the second parameter to this ad funk is the function that we actually want to run. And for now, I'm just going to put an empty function there. I can clean up this import comment, and we can save our file. Another thing I want to highlight here is the Visual Studio go extensions ability to auto format. So if we have some sloppy spacing in here or anything like that, as soon as we save, in my case, on a Mac Command S, we see that the inter automatically formats that code. I should also mention for these time zones, these are looked up from something called the Iana Time Zone database. I found a nice website called Nota time, which lists all the various time zones. So wherever you are out in the world, you can feel free to look through this table and set the time zone according to wherever you are, or whenever you want your Cron Job to fire. We also need to ensure that our cron job is started. So we've added the function, but we explicitly need to call cronjob dot SAT. And then if we were to leave this, when we execute the main function, it would just run here and then exit. So we do need to block the main thread to keep the go process running so that we'll keep this cron job live after we start it, and the cron will continue to run until each day when it hits that 8:00 A.M. Timeslot. In this lesson, we began writing our main dot go file. We learned that every go file begins with a package declaration. Then usually there's some imports and then your functions. In this case, it's a bit of a special case. The main dot go file always has a main function. And we began writing the body of our main function. We loaded the Vienna time zone, and we used that location time zone in our cron job, and we set up the Cron job to fire at 8:00 A.M. Every day. We saw a nice feature of how the visual studio code format can automatically fix and format the indentation on our code. And we also discussed all the various possibilities you can pass into this load location function so that you can set the time zone that you want. The next lesson, we'll get into building an HTTP utility function, so we can start looking at how we're going to call the API URLs that we eventually add in the function body here of our Crown job. 8. Building a Generic HTTP Utility Function: Last lesson, we started scaffolding our main dot go file and setting up the cron job that could fire off our various tasks at 8:00 A.M. In the Vienna time zone. Now we're going to write a generic HTP function that will be used for both the allergy API calls as well as sending our slack message. Now, this function is rather complex and does make use of G's generic functionalities. I won't go into too much detail as to how the function was constructed. I have a separate blog post that goes into the details of its implementation, and you're more than welcome to read about that separately. To get started with this function, we'll create a new folder called Utils and a new file called make http request dot go. The package here will be Utils. And for our function, make HDP request. That takes a generic type. Then we'll pass the full URL, which is a string, the HTTP method we expect, which is also a string. Any headers that the call needs, which is a string and string map. The query parameters which are of type RL values, the body, which is IO reader, and the response type T, and we'll return that type or an error. The first thing we'll do within this function is to initialize our HTTP client. And we'll convert the string URL to a full URL object with Rl dot pars with that full URL. If the error is not NIL, then we will just return our response type and the error. If the method is G, We need to append any of the query parameters that were passed. So first, I'll get the query from the URL object, and we need to loop through those key value pairs within the query parameters. And we will set the key and value. For each parameter. And then we'll set the raw query to the fool encoded built query that we've built here. We also can set the body and that's by making a new request, passing the method, the string version of the URL and the body. If the error is not null, again, we return the response type and the error. And now that we have our request, we need to set the headers. As for the key and value of those headers, we will set the header value. And finally, we can do their actual request, and that is quite literally with the client dot do function. Again, if the error is not NIL, we return that response type and the error. We'll also check if the response itself is NIL we'll return the response type and an error saying that calling the URL returned an empty response, and we want to pass in that string version of the URL object. Otherwise, if we continue on here, we want to read the body. And again, if the error is not nill, return that response type error defer the closing of that body. And we also want to check if the status code is not the o status code. Response type, and we'll also make a custom error here. Similar formatting here. You can throw in a new line with the status code, another new line with the actual response data. And then we want to put in the URL string, the actual status code, and that response data. Now, if we've gotten this far, we can finally unmarshal the response data into the response object, which is type T that we expect. So we'll declare the response object, and we're going to try and unmarshal the response data into the response object. If the error is not new, turn the response type and the error itself. Finally, we are safe to return the full response object with a NL error. Now when I save here, the visual studio go extension will automatically import all those libraries we use. So IO, the HDP library, the URL library, strings, and so on. So with a little bit of effort, we have a very powerful generic function, and I like to use this function for nearly any standard rest HTTP call that I need to make. As we've seen, it's implemented so that it can work for get put, post, and all sorts of HTTP verbs. So with this powerful function now implemented, in the next lesson, we will actually get to calling the allergy API. We'll also define the types that will pass as this generic type. So we know that the JCN response will be serialized properly, and then we can consume it further down in our G code. 9. Calling and Parsing the Allergy API: Last lesson, we built this rather powerful generic HDP request function, and now in this lesson, we'll make use of it calling the allergy API from the Medicine University of Vienna. So to get started, we'll make a new folder. Called Allergy API and a new go file Allergy underscore api dot go. And this package is the same as the folder and file name, Allergy underscore API. Now, the first thing we need to do is define the types that we will pass into our generic function here so that the JSON here, when it's unmarshaled, knows the type it should unmarshal into. And looking back into those two separate API calls, we have our hourly load data, which has kind of this overarching structure with the success and result keys, and then within the result, we have another object, which has a total, some sort of personalized option, and then most important for us, the hourly array, which is an array of integers. For the get current chart data, we also have a similar structure. We have a large object which has the success and result keys. But then the results here is directly an array with a repeating object that has date current, average season and date time. And once again, most important for us here is the average. We're going to take this endpoint for the historical average portion of the message that we send out. So let's get started with the hourly load data types. So first, I'm going to define the response, and that has a success and result key here. So I'm going to define this as hourly load response. This is a Strup and we have our success, which is an int, and the JSON identifier here is lowercase success. And the result, which we need to define a new type, a new struct for this nested type, I'm going to call it hourly load result. And we also need to define the JSON key that is lowercase result. Now since this type uses this other struct, I typically like to put the type above here. And what do we have? We have the total and the hourly total is an int. JSON lowercase total and hourly is a slice of ins, and the JSON is hourly. And when I save here again, the Linters formatting everything nice and neat into columns. Now we can move on to the chart data. So similarly, we have the Chart data response. And let's take a look at this. Also, success and result keys. So success is an int. And the result or I should say, results plural is in array, and this type will need to be another custom type, which I'll call current chart data result. And we also need to specify the JSON key. I'll take this and define the type here. And all we really need is the date, which for now will take as a string and the average, which is a float. So we have date and the average I can save this, and that should be all we need for custom typings. Now we'll write a separate function for each API endpoint. The first one, we can call get hourly load data. And the second, we will call get current chart data. Now for both of these, we expect that they return a string pointer and an error. And this string will be ultimately the message that will pass on to our slack messenger. And we know the goal here is to call our generic function here. So we'll have a response and an error from can import this Utils package that we've just created in the previous lesson and the make HTTP request. Now we don't need to specify the type explicitly. Golan will infer that for us. And for the URL, we have it here. That's this index dot pHP. We can put that in. We know this will be a get request. We don't need to pass any headers. We will need to pass some query parameters. There's no body that we need to pass and the response type is the hourly load response. And if the error not NL we'll return a Nil string and the error for now, let's move into these query parameters. This type is the URL values. And we can add all the parameters we need. So there's quite a few here for this hourly. We've got an EID, an action type can provide a zip country, and so on. So we'll provide all those query parameters. So we're sure that our Get request functions exactly as it does in the browser. So we're going to add this EID The type is ZIP. The ZIP is 6,800. The country is Austria A T. And there's also this flag for the Cure JSON, and that is one. Can import this URL package. And now let's move on to formatting and building this string that we want to return. So from the data here, we have the full array of the hourly load expected. And so what we'll do is loop over this array, build an average, and then create a string that mentions what this average is. So we'll first define some average load. And then we're going to loop at the range which we know is within the hourly part of the result. So that's just plus equals the hour value, and then the average load becomes the average load divided by the length of all the hours. One other thing I've noticed from looking around the API is that they're doing some sort of normalization of the data. It's not shown here, but these numbers can go as high as eight or nine, but they only show a ranking 0-4 on the actual UI of the website. So for now to kind of mimic that functionality, we are going to just divide the average, we'll call it a scaled average load, and that will just be this average load divided by two. The formatted message that we'll send, we'll take the format package, sprint F, and we'll say but the average pollen load for today is scaled average, and we will return that formatted message and no error. The implementation for get current chart data is quite similar. I'm actually going to copy all of this, and then we can make changes accordingly. And I've made a small mistake here. This is the action key, and the EID is, in fact, app interface. You can add that in here. Now, here, in this case, the action is, as we can see here in the URL is get current chart data. The whole ID, this is actually for the type of plant. In this case, I've made it for my allergy, which is for grasses or hay. We can still pass the zip. We only care about the season data, and that's the flag for the season data. And we also want that pure JCN flag to be activated. The URL is the same. The method is the same. We still pass, of course, the query parameters, and now we just need to change the response type, which is the current chart data response type. Now, of course, G will complain here because the shape of the response is different. So what we want to do for this method is to loop at all these results until we find a matching date, and then we want to print the historical average for the current date. I'm going to define a variable called current YY MMDD. That's time dot now, and then we need to format it using the Go style of formatting strings. We'll define an average historical which will initialize a zero and then we'll loop over those results and find the matching date. So if the result date equals this current date, then we know that the average historical is that result average. And similarly, as we did above, we should create a scaled average And we'll take a int cast of that since we only want zero through four, and we're just going to round the average historical found divided by two, and then we can create our formatted message. In this case, we'll say, historically, the average pollen load for today is and we'll pass in that scaled average. We save the inter, we'll import those packages, and we should be all set. So in this lesson, we got started thinking about how we can call these two endpoints and deserialize the JSON that they return. We started off by looking at the structure of the JSON and defining the needed types and the needed parts of that JSON that we need to use for building our messages. We then started writing the actual functions that call the endpoint, making use of our make HTP request that we built in the previous lesson. Once we get the response, we do a little bit of calculation and formatting. And for each function, we ultimately return the formatted string, which we'll be able to send through Slack. In the next lesson, we'll take this return string and send it through Slack via webhooks. 10. Creating a Slack App and Messaging Function: The last lesson, we defined some types and wrote two functions to call the Allergy API, do some calculations, and ultimately return a nicely formatted string, which we said we would then forward on to Slack so we could send the message via Slack. In order to send our messages via Slack, we'll first need a Slack app. Then we'll need to activate incoming webhooks for that app, and then finally, we'll have a URL that we can actually post to send the message. We'll then come back here into the code and write another utility function to send any message to our Slack app. To get started creating a Slack app, head to api dotslaq.com and go up to here and click your apps, and you'll need to sign in to get to your apps. I will sign in here with Google. And I want my company's Full Stack Craft account. And we can just click Cancel here. Now that we're logged in, we can head back to api dotslag.com. And then click your apps. Once we are on the app listing page, click Create New App. And we want to create our app from scratch. I'll call mine the Allergy Cron Bot. And again, I want it in my own Fullstack Craft Works. On the resulting page, we then want to add incoming webhooks. And we just want to toggle this switch to the on position, and we'll see some code appear here. Now, we want to add a new webhook, so we'll click here, and we need to pick a channel to allow it to post to. For now, I will pick General and click AAO. If we have a Slack app already open, whether on the web or the desktop version, we should see that the integration was added. So here in general, I see that I've added my allergy crown bot. Back in the web UI, you can see that Slack has created a webhook URL for us, and so we can copy this right away. Alternatively, you can copy their example here. This is a hello world example. And right in the terminal here, I will paste in this example, should fire off without a problem. And if we go over to Slack, we do indeed see that Hello World message from our brand new Slack app. For now, we'll just copy the URL itself and head into our G project. We'll create a utility function that we can send a message to. So I'll call this file, send Slack message. This is package Utils and the function here, send Slack message. We'll just accept a message, which is a string, and it will return an error, if any. All we need to do here is create a body, and we're going to use the JSON package and marshal a string string map with that text parameter, and that will be the message that we pass into the function. If the error is not Nil, we'll return that error. Then we can use our make HTP request. That's going to be our URL. We're going to post to that. The headers, we don't have to provide query parameters. We can also leave as Nil. We create a buffer from our body, and we can just have an empty string as our type. We don't really get any response. We would just get a 200 code when it's successful. So there's nothing we need to parse. And from here, we can return Nil. And if I save this file, the Linter will import the JSON package, and we should be all set. Now, this URL is technically a secret, since if anybody got ahold of this, they could spam your slack channel indefinitely. For now, we'll leave it hard coded, but in a later lesson, we'll be collecting all of these hard coded values and putting them into an environment file. So to sum up in this lesson, we created a new Slack app, and we activated incoming webhooks. And after selecting a channel that we wanted to message, Slack generated a URL that we can post to that would ultimately forward the message onto that channel. Back in our G code, we wrote a concise send Slack message UTIL that we can then use to forward on what we parse from the allergy API. In the next lesson, we'll combine all of the functions that we wrote back in main dot go, and then we'll be ready for the very first time to run our application. 11. Completing the Cron Job: Last lesson, we created a Slack app and this send Slack message utility function that can actually send the messages to Slack that are generated in our allergy API functions. This lesson, we'll finally combine all these functions that we've wrote in main dot go. So within this Cron function, I'm going to call our G hourly load data and get current chart data and then forward that combined message on to Slack. So we've got our daily average message. And that comes from the allergy API. Get hourly load data. If the error is not nill, we're going to panic. And we'll also get the historical average. And likewise, if the air is not nill, then we'll panic. And then the actual slack message we'll send is going to combine both of those messages, and we'll just put a new line character between the two. With that combined, we can call our Send Slack message function with that slack message. And again, if the error is not nill, we will panic should be Utils. Can import that. And just for now for debugging, I am going to log out that we successfully sent the Slack message, and we'll add in the actual Slack message. Can save that, and we should be all set to go. So as a quick review for this lesson, we hopped back into our main dot go file finally, and we called the Get hourly load data and also the G current chart data to have both a daily average and historical average message. We combined those with a new line, and then we called our New Send Slack message function, which ultimately completes the full functional flow of our application. In the next lesson, we will modify the cron job time to fire more or less immediately, and then we will run our application. 12. Running the Application: The last lesson, we completed the body of the function that will actually fire at our specified cron time. Now, since we're ready to actually run our application, we should update this to the next immediate hour and minute so we can quickly see the cron job firing and we can test the actual body here to see that everything's working. So as of this current recording time, it is 1209, so I'm going to bump this up to 1210, and then we can call our function with go run main dot go. And we should expect right as the clock gets to 1210, these APIs to be called and our slack message to be sent. So, indeed, we see just a tick after 1210. We get the following slack message that's sent. The average pollen load for today is one, and historically, the average pollen load for today is two. We can also see over here in our Slack app that we indeed get the same exact message. So our application is working exactly as we expect. So so far, what we've built in this course follows the technical requirements that we specified in terms of the flow and how the app works. It gets the data from the API, it converts it to a nice human readable string, and then sends that message via Slack. However, there's still a lot of optimizations we can make to make this more of an enterprise grade application. So in the coming lessons, we'll look at things like removing these strings and putting them into an environment, as well as things that really should be left to an environment file like the Slack URL and also the endpoint of the allergy API. Once we do that, we'll move on to looking at testing and then even implementing an automated CICD pipeline. 13. Writing Tests: Last lesson, we ran our application, and we saw that indeed everything is working as expected. Now, that's all fine and good, but what if the API changes or something weird happens in our application causing it to break? We probably want to know if any of our functions are broken without having to run the application in a complete production environment. So in this lesson, we will write tests for each of the functions that we've written. Specifically the Send Slack message function and also the two functions within the allergy API, the get hourly load data and the get current chart data. We won't explicitly write a test for M HTP request because this function is actually called by the other three that we'll write tests for. So by proxy, if we write a test for these three functions, we'll also, in turn, by proxy be testing the make HTP request function. To get started writing our tests, we'll make a folder called tests. And I'll make two files, one for each of the modules or functions that we'll be testing. So we're going to have our allergy API test, and we'll also have our Send Slack message test. And note four, go to recognize these files as actual tests. They do have to end with the underscore test dot go suffix. So within each of these, this is our package test. We know that we're going to need the testing module, the built in testing module from G. And also for the function name, there's a rule that the name has to start with test. So we need both the underscore test suffix here and also the function name to start with the word capitalized test for go to fully recognize both the file and the functions within the files for it to be a valid test. And so here I will call it test Allergy API, and we do have to pass in a pointer to that testing module. Leave it empty for now and do the same here. For our Slack message test, this is package tests. Import the testing module. And I'll call this test send Slack message, and we pass in a pointer to that testing module. Now to actually test our functions, we will call the G hourly load data and the chart data functions. And now we call our function as we would anywhere else in our code. So I'll first call this get hourly load data, and then we'll do various checks on the error and the message to complete our test. So if the error is not new, then we want to use the T dot f function, and we can say error getting hourly load data. We can pass in the actual error. You should also check if the message is Nil. That is also an error case. And in this case, we'll just say error getting hourly load data. Message is NIL and we can also check if the message itself is an empty string. That's probably also not very good result of our function. So error getting hourly load data. Message is empty. And nearly the same for the get current chart data. Got our message and our error and the same checks. So if the error is not nil can actually copy this. Just be sure to change the message here. So it's clear when we run our tests. We'll also check if the message is Nil or the message is empty, and I will just replace this hourly load string with current chart data. Now, within the Send Slack message test, we have only an error returned from that Util function, and I'm just going to call the Send Slack message with test message. If the error is not equal to Nil, then we'll say error sending slack message, and we'll put in the actual error now we'll use the built in go test command to run these tests. So I like to pass a few flags when I run my test. The first one is P, which sets the number of parallel tests to one. In other words, that would run your tests in serial. And I like this format because your tests will always run in the same order, and you always get an expected outcome for how your tests run. So over time, you can get used to how that output looks. If you have a very, very large code base, of course, you could consider modifying this to run multiple tests in parallel. I also add the V flag, which enables verbose output. And this is useful that in the case something does go wrong, you get more output and you can find the error faster. We also do need to pass the folder where we want to run our tests. In this case, it is the tests folder. So together, the command is go test P one and then for the Verbose and then the tests folder. And if we're any good at our jobs, when we run this command, we would hope to see that all tests pass. And indeed, both of our tests, the test allergy API and test sens slack message, both passed, and our tests are looking good. So as a quick recap, we wrote the tests for most of the functions that we wrote in our code base. We put them in the test folder, and we discussed how to be a valid test for G, G needs to see both the underscore test at the end and the function name within the test file to start with capitalized test. We then wrote our tests. We ran them with the G test command with a few additional flags, and we saw that all our tests pass. The next lesson, we'll look at how we can dockerize our application. 14. Dockerize the Application: Last lesson, we wrote tests for our application, and we saw that indeed the tests were passing. In this lesson, we'll look at how we can dockerize our application. Because GO compiles to a single binary, there isn't too many complex steps that we need to make for our GO application to run smoothly in Docker. We'll need to define two files, that is the Docker file itself, and then the Docker compose file, which will allow us to run our container with a tool like Docker Compose, where we can manage and orchestrate our containers. So I'll get started with the Docker file, and I'll put that right here in the root of our project. That's simply Docker file. And I'm going to start with the Gang alpine container, and I'll take the 1.2 Golang container. So that's from Golang 1.2 oh Alpine. Then we're going to define the Wd as app, and that's a common practice so that we're not building or creating files in the actual root of this container, but kind of defining our own workspace typically is taken just as app in this container, and that's where we'll do the actual building. So then we want to copy everything that we have here in our code base to that app folder, and that's just dot to dot. Then we'll actually build our application. So we'll do Go Build O, and we'll call it Allergy Cron and then to run this, as mentioned, G will create a single binary, and we just need to run that binary. So already this Dockerfile would be enough to create a container that we can build and then run, but to make it more friendly with Docker Compose, that we can run it immediately with Docker Compose, we'll also define a Docker Compose dot Yamofle. So just like the Dockerfile, I'll put right in the root here, the Docker Compose Dot Yamofle and I'll use the latest version for Docker Compose. That's 3.9. Then we define our services. We have the allergy cron, which is our only service. And then we'll define the build context. It's just here, which means Docker Compose will look for this Docker file right in our root and use that Dockerfile. Then we can also define the restart policy, and I'm going to define it with stop less stop value. So this means if our container crashes for any reason, Docker will detect that and restart the container for us. Now to see if we've configured everything properly in these two files, we can both build and then run our container. So first, to build the container, I'm going to use the Docker Compose command, and we want build. And since it's our very first time building our application kind of as a double safety check, I'll also issue no case, and that means that anything needed to build the container will be freshly downloaded from the Internet. We're not going to use any local files here. Once that's done, then we can also run Docker Compose. And in this case, we want up and I'll also pass the D flag, which is the detached flag, and that means that Docker will start up our container in the background and give us back our terminal. If we don't pass that D flag, we would start seeing the logs of our container appended directly, and leaving or exiting the terminal environment would also exit from our container. So we pass that D so that the container will stay up and running in the background. So if everything works properly, we should see a nice green done here at the end of all the build output. And we can check to see that the container is actually running with the Docker PsA command. And we do indeed see our allergy Cron running created 18 seconds ago up 15 seconds. And so we've successfully created and ran our go application in a Docker container. As a quick recap, we define both a Docker file and a Docker compose file to allow our application to be run with the Docker Compose command. We both built and then ran our application in the background, and we do see, indeed, our application is running in its container on our local system. In the next lesson, we'll look briefly at restarting our container with minimal downtime. 15. Restarting the Container with Minimal Downtime: The last lesson, we created the two configuration files necessary to both create a Docker container itself and then run that Docker container with Docker Compose. In this lesson, we'll look quickly at how we can replace a running container with minimal downtime. And really, it comes down to rebuilding the application and replacing it. And again, that is using this Docker Compose command. We're going to build again with no cache. And issue the up command detached and importantly with force recreate. We pass this flag so that we're sure Docker will always replace the existing container with the newly built one. So if we issue this command, we again see that it completes. And if I check our containers with PSA, we do indeed see the new version created 10 seconds ago up 7 seconds. So it was just replaced. Now, for a Cron Job application like this, this command is probably enough for your needs. But keep in mind, this is not a zero downtime method. This is just a minimal downtime method. You would have downtime for the short moments that it takes for Docker to replace the container. For a cron job application like this, it's just important to remember to not replace this container near the time of when your crown job runs. If you're doing something more complicated that needed a true zero downtime, you would need to do something more complex like making multiple containers, switching load balancers, and so on. But that type of strategy falls outside the scope of this course. But in summary, ultimately, if you know what your app is doing and when it's safe to replace that container, this method works fairly well for many types of applications. In the next lesson, we'll get back into some coding and improving the message that we send out via Slack. 16. Adding Fancy Formatting to the Slack Messages: In the last lesson, we briefly discussed how we could rebuild and replace a container with very minimal downtime. In this lesson, we'll get back into some code and look at how we can improve the formatting of the slack messages that we send out. So if we hop into our allergy API file, we can recall that we're sending two fairly plain slack messages. It might be nice to add a bit more color and maybe even a few emojis so that the sentiment of our message can be more rapidly and more interestingly displayed. So what we'll be doing is modifying this initial message. We're going to leave the historical message as it is. But for the actual real time data, we'll improve the formatting of that message. So down at the bottom here, I'm going to define a new function called format Allergy data. And it's just a lowercase function here because we don't need to export it. It'll just be used within this file. I'm going to pass in the scaled average load that we have that's an int, and we're going to return a string. What we currently have is our formatted message and it looks like this. The average pollen load for today and we pass in that scaled average load. But we're going to both prefix and suffix this message to make it a little more fancy. We can do that with a switch case and what I'm going to do is switch on the scaled average load. And for one, that's kind of a moderate case. And so we'll prefix it with this yellow circle emoji and put Okay, then a space, and then the existing message we have. And we maybe want to put caps that it's a low or a moderate level. Then in the case that it is two, We can bump up the color to kind of orange warning color and say something like watch out again with our formatted message. And in this case, this is medium, and we'll end it with that orange emoji. Then four or three. We'll put red and put warning again with that formatted message and reference that this is high and ended with the red emoji. Now, if it's the highest level, which is four, then we're going to put three of these red emojis and put alert with our formatted message. And put very high stress these higher cases here. And we'll also conclude this with three of those red emogi circles. Then if the case, if it is zero, we'll fall through, and we'll also take default and return this green emogi then say nice with the formatted message, the none. Just to make the punctuation consistent here, and we can close out the bracket of our switchcase. So now, instead of our format here, we will get rid of this since it's already down in our new function and call the format allergy data function and pass in this scaled average load. Now, if we save this, we can test our application. So right now it's 318. Going to bump up our cron to 319. So that will be 15 and 19, I believe. And we can issue go run main Dotco. And indeed, just after 319 here, we get a copy of what we've sent over Slack with our nice new message. And today, it's kind of hot and dry, so I have to look out a little bit. The level is high today. So as a review of this lesson, we went into our allergy API file, and we replaced the first message that we send with this new format allergy data function. And we added some extra additional text here, as well as some emojis that better reflect the sentiment of the information that we're sending out. And overall, I think it looks pretty nice and it's more easy to determine the nature of the message right away. So there's only a few lessons left. In the next lesson, we'll look at moving all these hard coded values like the URL and the Cron time zone, as well as the Cron interval, and put those in an M file. And once we've done that and ensured our application is still working, we're going to use CircleCI to build a CICD pipeline that will automatically test, build and deploy our application to Docker Hub. 17. Moving Secrets and Hardcoded Values to an env File: Last lesson, we wrote this format allergy data function to improve the formatting and make the slack messages a little bit more colorful. In this lesson, we're going to take all of these hard coded values, such as the API URL, as well as the cron location and cron schedule and put them into an environment file. So to get started, I'll create an N file, so dot N. And immediately, we want to create a Get Ignore file and add this N file to the G Ignore. This is a common practice, as typically you'll have at least one or possibly more secret values that you don't want to check in to your Git repository. So I'll create this Get Ignore and I'm just going to add the M file. So what do we actually have for our environment variables? We have the Allergy API root URL here. We also have our Slack web hook URL, and we have our Cron time zone as well as the schedule string. So let's add all of these to our environment file. So we've got our Slack webhook URL. Can paste that in. Got our allergy API URL root, I'll call it. That's this guy. Got our cron time zone. That's here. And the cron schedule. And that's this one. And while I'm here, I'll revert that back to the 8:00 A.M. Original value that we had. Now, for the Gang runtime to actually see or have these values in the environment, we need to make sure that this environment file is loaded into the runtime. To do that, we can use the popular Johogo dot Nv package. So that is github.com, Joho and go dot v. And we can retrieve this here. And right at the top of our main function, we can load that in. That returns an error, and it's just go dot N dot load. And we don't need to pass any parameters. It will look by default for our dot N file. Of course, we want to check if the error is not Nil then we're going to have a fatal message here. Error loading file. Now we need to actually take the value of these environment variables. That can be done with the s.gn call. So we've loaded them very first thing in our main, and they should be available then in our environment. So we've got OS GNV. And this will be the Cron time zone. And for this one, we've got OS GNN. This is Cron Schedule. And likewise here in the API OSG N you also have to add it below, I believe, or above and also in our slack function. This is OSG N. And our slack webhook URL. Now, we will also need to add this loading of environment in our test as well. Since now that we've refactored the code, the code that runs under these tests also expects to have those environment variables. So it's slightly different in the test because we're not in the root, we can't just call the default load, but we have to explicitly pass the path back to the environment in the root. I'll copy this into the top of both our tests. And we just need to pass explicitly that path. And import both the Godot end and log packages here. And same thing for the other test. Import those. And we've already defined error now, so it's just the normal equals. And that should do it for the tests. So to quickly review, we created a dot N file for all the various hard coded values that we were using around our application. We also created a get Ignore file to be sure that we're ignoring that dot N because there are indeed secrets. In this case, most important is this webhook URL. We don't want anybody getting at that or else they could post to our channel. We went through our code and replaced those hard coded values with their respective environment names. And we also added to our tests the way to load the environment variable. So in the case of our test explicitly passing the path to our end, which is in the higher folders. So we use the dot Unix notation to get to that root folder from our test folder. Ultimately, this is a very nice pattern because right in this M file, you see immediately some of the most important key values for how our app works, namely the time zone and schedule, but also if the URL were to change or if you were to find a different vendor, you could change this URL and write a different client for that API. Also, for example, if you were to change your webhook URL to a different channel or if the team changes, you could also quickly replace that here and then know that it's being used around your code base, wherever that may be. So now that we've dockerized our application, we have cleaned up the slack messages, and we've also cleaned up hard coded values. It's finally time in the next lesson for us to build a complete and full CICD pipeline to build our application and deploy it to Docker Hub. And we'll finally close out the course by pulling that newly created container from Docker Hub and running it one final time to see that everything is still working. 18. Creating a CI CD Pipeline with Circle CI: In the last lesson, we created an environment file to store all the hard coded values and secrets throughout our application and replaced those hard coded values with the names of the environment variables around the application. And that was really the final step for us to be ready to package and automate the way that we build our container. So in this lesson, we'll be creating a full CI CD pipeline that will build and then test our application with the tests that we've written. Then if those tests pass, it will deploy or upload our container to Docker Hub. Then in the end, we will pull that container and run it as a final test to see that everything is working. So to get started with CircleCI, we need to first make a folder that is Circle CI, and within this folder, a config dot yaml file. Now, the first thing you need to do in your config dot Yamal file, similar to the Docker compose is provide the version. Currently, that version or the latest version is 2.1. And we'll also list an orb. Now, orbs are prepackaged commands or jobs that are very common to execute. For example, they have a node orb. They have a slack orb, and that just saves you time instead of writing explicit bash commands to accomplish tasks, they have prepackaged things. For example, send Slack message, and you just have to pass the string message. You don't have to issue this curl statement or things like that. In our case, the only orb that we need is for the Docker hub, and that can be done by specifying Docker, and we'll take the latest version of that Docker orb, which is 2.2 0.0. All that's left now are two main parts of a config dot yaml file. There are both jobs and workflows. Jobs are various steps that you may use one or more times in your workflows. So you could think about them as individual building blocks, and workflows actually combine and say in what order and how those jobs should be run. So typically, workflows are listed higher up in the config dot yam. But since we're going step by step here, we'll first write the jobs, and then we'll write the workflows after. It makes a little bit more sense from a step by step perspective. So we'll start by defining our jobs. And for now, we really only have one job, and that is to test our application. So I will just call it test we have to specify a working directory. And in the Circle CI world, this repo is the special signifier for the local repository. And we're going to use a Docker image, and we're going to use the GO image. Then we can define our steps of what we actually want to be done within this job. So first, we're going to, of course, check out the code, and we can cache our go sum file. This will make subsequent builds faster. And we're going to put check some of our go s and we'll also install dependencies from that. Need to issue GGEt and then we'll save that cache. So that's in case later down the road, if we install new packages or change the versions of them, that'll be accurately reflected in this key that we've defined for our cache. And we also need to define what path that's in. So most of this was taken from an example that CircleCI has on their site for the recommended and best practices for a go application. And I'll put a link to that in the Lesson resources. Now we also need to build our environment file because we know our app can't run without those environment variables. So we'll have another run step here. The name is create dot N file. The command can use the pipe to do multiple steps or multiple commands. We're first going to create that file. Then we're going to use Echo to echo all the environment variables that we need. To escape into the CircleCI environment, we use this syntax, the brackets with the dollar sign, and we'll just take the same names that we have in our application. We'll see how to define those later in the CircleCI UI. And we'll just append to that new file. So I'm going to copy this a few times, and we know that we have the allergy API URL root. We also have our cron time zone, and we also have that cron schedule string and with that complete, then we are going to issue tests. So I'll just call this run tests. And the command we know from the previous lesson is go test. We'll set that parallel flag and also the Verbose flag, and we want to run that on the tests folder. So we've defined our single job test, and now we need to define the workflows. So the normal pattern, as I mentioned, is that the workflows do go above the jobs, so we'll hop back up here and also specify our workflows. So we also just have one workflow, and that will be the production workflow, and we need to specify the jobs. And we have our test job, and we're also going to filter on the branch. And we want only the main branch. And then the second job that we want is going to make use of this Docker orb here. And that's going to be Docker Publish. And the image will be both our username, and repo name, which we'll also define later in the CircleCI WebUI along with our other environment variables. So we'll get to that in a few moments. Now, we also need to define the order of these jobs. If we don't specify any order, CircleCI, we'll just run them in parallel. That can be useful depending on what you're doing. But of course, in our case, we do want to make sure those tests pass before we publish to our container. The way to do that is with the requires directive, and we require, of course, that the test job completes. And we also want to filter on the main branch. Now, there's a lot of Yamil code in this file, and it's not clear if we have any syntax errors or issues, but luckily, CircleCI provides a CLI tool where we can check this config file. I'll add in the lesson resources the link to their official documentation on how you can install that. I already have it installed on my system and the way to check the config is with CircleCI Config Validate. We can see here I've forgotten looks like the semicolon here. And I'll re run that check, and I've also forgotten an S. And finally, it looks like we have a valid config here. So we get the is valid. So we can already see it's quite practical for finding any sort of typos or formatting issues in our config file. Now, at this point in your code, it would probably make sense to branch off and create a developed branch. First, of course, requiring that you've initialized the Git repository and assuming perhaps that you're still on the main or master branch, then you would, of course, issue, checkout B develop and commit everything to that branch. Then what you would do is merge when ready, merge to your main or master branch, and then that would kick off this case, it would have to be the main branch or else CircleCI wouldn't do anything. It's waiting for commits to this main branch as we've specified. But if you have, for example, taken the master naming convention, you would have to change this to master for CircleCI to do anything on that branch. In my case, I'm already here in the course specific repository, and I have a custom branch name. So for now, I'm going to leave this as is, and we're going to hop into both Docker Hub and CircleCI web applications and configure what we need to there. So within doctor Hubb, sign in or create an account. If you don't have one, they're free. And we'll just click Create repository here, and I will call mine Allergy Cron. We can just click Create here. And now we have a repository that we can push to. Then we're going to head over to circleci.com and click GT application. And on the login page here, since the repository in my case, is on Github, I'm going to log in with GitHub. If you've decided to follow along with Bitbucket for example, you can log in with Bitbucket. And so I'm in a bunch of organizations, but the repository that we've been coding in is on my personal account. So I'll select that and you'll see all of your repositories in your GitHub profile. And, of course, I want this go for real world applications course, and we can click Setup Project. So before CircleCI can find Arc fig file, I have to go back here in the code and commit this branch. So I'll add everything. I'll commit something like CircleCI Config finished. I'll push that I have to set lesson name with the branch name. Then here I should be able to specify the Lesson 18 branch, and we see that CircleCI will even look through our code and find that config Yamofle. So it's really, really a nice service. I really enjoy using CircleCI. They make it very easy. So in your case, depending on where you've pushed, if you've pushed it to the developed branch, if you've pushed it to the main branch or the master branch, you could specify that here, and hopefully CircleCI would find your config dot YamoFle. So once it's found that, you can just click Setup project. And it will even try to kick off the very first workflow. But of course, since the commit was to the Lesson 18 branch, it sees in the config, Okay, there's nothing to do for Lesson 18, and it will just say no workflow. So now that we've got our CircleCI project set up, we should add all those environment variables to the actual CircleCI environment. To do that, we can go up here to project settings and over here to environment variables. And we can add our key value environment variable pairs. And we know we have our Slack webhook URL. And we can go on through adding all the variables that we know we need for our application. Got time zone. I've got the cron schedule. And now we also need to add a few for our Docker Hub integration. So we need to add the Docker username, the Docker login, which are actually one and the same, the Docker password, and the Docker repo name. So in my case, my Docker Hub is our company account, Full Stack Craft. So I'll be using that for both the Docker user name. And also the Docker login environment variables. The Docker repo name is the name you provided for your repo. And in my case, that's Allergy Cron. And finally, the Docker password. Of course, I'm not going to show that here. So we've now defined all the environment variables that we need to run CICD pipeline. Back in the code, since in my case, I already have a main branch that I don't want to mess up, I'm going to create a separate branch called Pipeline and also update that branch filter in the config dot Yam file, and then we'll be able to test the pipeline. So I'll just modify this to Pipeline. And remember, of course, in your case, you can leave this to Maine or whatever branch you want your CICD pipeline to fire on. So I'm going to create that new branch with G checkout B Pipeline. Can indeed see we're on the pipeline branch. Going to add everything. And I'll add a message Something like custom pipeline branch, and we can push. Now back in CircleCI, we should see the production workflow does indeed kick off because CircleCI sees that pipeline branch filter. We can click into here to see both of our jobs. So we have our test job and our Docker published job. And even within the jobs themselves, you can see all the steps and the output. So it looks like our tests have passed. This is the familiar output that we've seen in the previous lesson. Then, of course, back in the workflow, it's going to move on to that Docker publish, and we'll see how this goes. Looks like that was successful, as well. And indeed, on Docker Hub, we do see that it was published a few seconds ago, so our CICD pipeline worked flawlessly. Now, although our build and workflow appears to be working, if we were actually to pull our docker container and try and run it, we would see that G complains that it can't find the environment variable. We've forgotten the keystep in our CICD pipeline to persist that environment variable between these two jobs. And so to do that here at the end of our test job, we can specify this command persist to workspace and we want to specify that the root is here, and we're just going to persist that dot m file that we create here. And then up in our Docker publish command, we need to specify that when we are publishing or building that container, we want to attach at this current root. And then when Docker's building, it will have that environment file in its build workspace. So one very important thing that I want to stress here is that this is a little bit of a security risk. So please note here, in this case, this is a publicly listed container, and the file will be inside the container. So while this is fine for public variables like the Cron time zone and the cron schedule string, it's not okay for secrets like our slack webhook. In that case, I would suggest that you create a separate file and pass that in when running your Docker container, wherever that may be. However, these secret managing specifics tend to be very different depending from organization to organization, and so I'll leave that outside the scope of this course. For now, we'll just illustrate how we can include this file into our CICD. But just please note that it is a potential security issue. One final small change we need to make here in our configured amo file is to escape this cron schedule environment variable. Because we have these asterisks here, when they're escaped by CircleCI, we get some weird behavior with those characters. So to remedy that, all we need to do is wrap this variable in double quotes, and that will fix the problem of echoing that value into our end file. So as a final test for our container, I am going to change this crown schedule to something in the next few minutes, and then we can pull our container, run it, and ensure that everything is firing and working, indeed, at the time that we set our cron schedule. So it's about 4:18 right now. So I'm going to delete this and add it back in. Let's do for 225. So that's the 25th minute, 14th hour, and then the asterisks for all the others. Can add that back in. And back in the project here, I will re run the last workflow to ensure that the container is rebuilt with that new variable. So just waiting here until the pipeline completes, and then we can hop into doctor Hub, get our container ID, and run that container. So the pipeline has completed, and if we hop over here, to Docker Hub, we see we've got our new image just published. We click on that. We can get the full ID. Going to copy that. Then to run it, we can issue Docker Run. Then we want detached and that full ID of the container. It's going to download for us and start running it. So we can check that it's running with P A. We see it's running and no logs yet, but we do expect at 2:25, we should get a logged copy of our Slack message, and, of course, the Slack message itself. Okay, so 225 just passed. Let's check out our slack here. Indeed, we do get that message. So today it's a bit rainy. The pollen load is a bit lower than the average. Some good stuff for me to know. And if we go back in here to our docker logs, we get that copy of the message that's sent out all the way back from that log message we put in main dot go quite a few lessons back. So, congratulations. You have reached the end of the last technical coding lesson for this course. There's only one lesson left, and that is the sort of Outro and recap lesson going over what we covered in this course, as well as discussing where you can find the code, the book, and any other additional resources for the course. 19. Outro: So congratulations. That's it. That's the end of this go for real world applications course. I hope you enjoyed taking it as much as I enjoyed making it. Just a reminder, there is a PDF book version of this course. I'll add the link to that in the Lesson resources. There's also the GitHub repository for this course where each lesson corresponds to everything we did in that lesson. They're named by branch. So that's all I've got from my side. Enjoy writing G code, and I will catch you all next time. O.