Getting Data from a Remote Source with Ruby: Hunter-Gatherers Still Need Their Tools
At this point in my Flatiron School Software Development curriculum, comfort zones are seemingly born for the sole purpose of being destroyed. This, though I’ve learned, is a good thing. For my first CLI (Command Line Interface) project I would be building my own functional program essentially from scratch. On top of that, I had to implement something called an API. And just like that my comfort zone was murdered, and I was on what Kenny Loggins likes to call the “Highway to the Danger Zone.” But, much like everything else on this new journey, I had to take it one step at a time. And if you’re reading this, hopefully it can quell any fear or insecurity regarding gathering data for your programs from a remote source like an API. So let’s break it down.
API is an acronym standing for Application Programming Interface, which essentially means it acts as an intermediary between programming applications. If you’re familiar with the colloquial “middle man,” think of it like that. An API’s central function is to serve as a collection of data that any programmer can use. My classmates and I were instructed to scour websites like RapidAPI (the one I ended up using) to find an API that suited our CLI program’s needs. I wanted to make a simple, yet reliable program that could pull information from IMDB.com, which is a website dedicated to detailing essentially every film ever made. I wanted to create a sort of database for the director Sergio Leone, who pioneered the “Spaghetti Western” genre with five films throughout the 1960s and 1970s. After some digging I found a free (some aren’t) API that allows a user to search through IMDB’s extensive database of films using only a parameter of a film’s name. The only required steps from this point were to create a RapidAPI account and subscribe to this particular one to generate an API key, which essentially grants a user personal access to the information contained within the API. The hunt for the API I was going to use was successful — all I had to was grab the code they provided on the right. But that’s just step one.
So the first thing I needed to do is change the code snippet they were offering me into the language I would be using in my program. You can do this by clicking on the drop down menu right underneath the words “Code Snippets.”
I was going to be working in Ruby so I chose this option, however you may notice there are two subcategories under Ruby: “net::http” and “unirest.” The “net::http” category means that it will utilize the “net/http” gem to create a new request for the API. And for “unirest” the program will utilize the “unirest” gem to create a new request as well, but since “net::http” comes with Ruby upon downloading the language and I was more familiar with it, I chose this option. So when back at the API I typed in my required parameter, in this case a film name, to fill in the URL address for the film’s location on IMDB’s website.
You can see here the API’s preset code is taking our web address, or URL, and using the Ruby gems “uri” converts the address into a URI, or Uniform Resource Identifier, and sets it to the variable “url.” In order for the “net/http” gem to function properly and create a request for the IMDB website, it needs to be a URI.
Note: The code here is also utilizing the “openssl” gem to which functions as a way to to create a request that interacts with “https” or secure websites. IMDB happens to be one of these websites as you can note by its URL : “https://www.imdb.com/”. If you would like to learn more about SSL/TLS certificates, keys, and the “openssl” gem, the following website breaks it down fully: “https://www.digicert.com/kb/ssl-support/openssl-quick-reference-guide.htm”.
So now that I had the correct URL for the film from which I wanted to gather data, I could look down and see the following code to generate a new request using the “net/http” gem:
You can also see here that “rapidapi key” is set to “SIGN-UP-FOR-KEY.” Before creating a RapidAPI account and subscribing to this particular API, this is the default text. Your personal key will be generated upon creating that account and subscribing and will be filled out in place of the default text. The above code “Get”s, or creates, the new request for the “Unofficial IMDB API” and sets it equal to the variable “request.” Just under that you can see the following code which has the variable “http” from the above “openssl” gem interaction take the method of “.request” with and argument of the variable request from the above code. The code then “puts” out the response taking the method of “.read_body.”
After copy-pasting the code into my own program file, I placed a “binding.pry” from the all-powerful Pry gem, which I required at the top of the file with the rest of the gems already being used, in between the “response” variable and the “puts.” While in Pry, I then typed the “puts” line to see what it would return.
You may have to open the above photo to read it more clearly, but to me it looked intimidating. It was a great deal of information in a hash data structure with keys associated with most films that pointed to the values of my searched film, “The Good, The Bad, and the Ugly.” Even though it seemed like this was beyond my reach, I knew there was a way to parse this information into something at the most more accessible, and at the very least, more readable. This is where the JSON gem comes in. Clearly I couldn’t keep the the line of code “puts response.read_body” because it returned a giant hash. I needed something more manageable. So I typed “require ‘json’” at the top of my file and put what I knew to work about making this response body a more readable and, hopefully, more functional for my program. I not only set the “response.read_body” method equal to a variable “body” and then utilized the JSON gem to parse this information with the “.parse” method, but also used the JSON function “symbolize names” to make the keys in the hash more visible. I then set this entire line of code equal to a variable “ind_film_hash,” commented out my old “binding.pry,” and then placed a new one right underneath the “ind_film_hash” variable so that I could see what that variable returned in Pry.
I felt like I had toppled a giant. I could now easily read and access the results from passing the response body from the “net/http” request through the JSON “.parse” method. I could then be able create a “Films” class and simply initialize that class with all of these key value pairs to eventually display information from the films. Then through my code interacting with the API, I could instantiate a new instance of the Film class with all this information. But there was one problem. This is only the one result from one request to the API. How was I going to get my program to get the other four films? My first thought was that my API key granted me access for this one request, like a key opens one door, but if you need to open other doors that grant access to other information, you need more keys. After Googling that idea and feeling like even Google laughed me off the webpage, I had to sit and think about what I had learned fundamentally about Ruby.
I had hunted the information about the film, I was able to gather the information about the film and “puts” it out to my local machine, but how could I get all five films at once? The code that generated a request and response from the API was just data, too. My mind drifted to a loosely remembered quote: “Ruby is a pure language. The only thing that you really are ever doing is calling methods on objects.” From there I came up with this: each of the five film URLs could be URIs themselves and I could instantiate new instances of the Films class by taking all the film URLs, placing them into an array, set that array equal to a constant, and then iterate through the array of films instantiating a new Film class object for each instance of a film. My mind had gone blank when confronted with a new task that was seemingly insurmountable. We, as programmers, cannot forget the most basic of tools like the iterator when building programs. It would be as if someone showed up to a construction site to build a new home and didn’t bring a hammer. It was a hugely cathartic moment: I know now that any new obstacle placed in front of me will only grow in size and complexity but it won’t be something that I cannot act upon. The tools we possess may be simple, but as you can see by this example, they are extremely powerful. In the end my API class code ended up looking like the screen shot below, where I iterated through the FILM_URLS array with the iterator “.each” to instantiate new instances of the class Films so that each film could eventually be displayed with the data from each of their individual hashes.
I hope this demystifies gathering data from remote sources somewhat. Remember you already have everything you need, just don’t forget your tool belt on the way out the door.