Lessons From Album Tags: Part 1

Greetings friends, well-wishers, and fellow learners! For those unaware I have been working at the Application Performance Management SaaS company New Relic for the past six months. Despite the blog-silence, I have kept to my word and have continued to learn something new every day! Definitely check out my repos for Node Projects, .NET Projects, and Ruby Projects as well as Album Tags and Counseling Book Tags to see some applications I’ve built during the learning process!

Today I am going to focus on one of those projects, Album Tags. I am now about nine months into the Album Tags application as I come up on the end of my first full year of software development. I wanted to write up a few posts describing some of the specific lessons I learned from building and maintaining this application over all these months.

From the start, Album Tags was a passion project. In a world of playlists and song-centric music collections, I did not have a fast, intuitive tool to manage my album collections, other than extensive Apple Notes files that were quickly lost and forgotten. In a way, I was lucky to pick this project so early in my learning path because I am the exact, target user. I had very specific feature demands for the application and I was able to easily confirm if my requirements were being met, or if there were usability improvements that could be made. I have also been able to continually apply new tools and approaches I learned along the way to streamline the user experience. I still use this application at least once a day to look through albums, add new albums, find music for friends, and continue to build the Album Tags database.

The first series of improvements I’d like to talk about here is the progress of the underlying Album Tags data structure. Of all the elements in the application, this area has seen the most frequent large-scale changes as the application evolved. At the start, Album Tags used a single MongoDB collection with the Album as the central data model. An album was modeled like this:

{
    “_id” : “MongoDB document ID” 
}, 
{  
    “albumId” : “numerical Apple Music album ID”,  
    “tags” : [“An array of tags”, “stored as strings”] 
}

This enabled me to retrieve albums from the database by the Apple Music Album ID, or by one or more tags associated with the album. Using this data structure, I built out the first Album Tags functionality: searching by tags. I used the Apple Music API to retrieve information about a specific album and parsed the data on the front end to populate a UI that showed things like band name, album name, release date, and album cover. I then pulled the tags from the database for the associated album, and allowed the user to search by selected tags to find other albums with matching tags. Even at this early stage, the application was already more powerful than any system of linked notes I had come up with to-date, and I was able to use it to share albums with my friends in new ways. (NOTE: the term ‘share albums’ at it is used in connection with Album Tags refers to sharing information about albums, not file sharing.)

As friends started to use the application however, I quickly realized a down side to this data model. It does not monitor who specifically created a tag, either for curation or for controlling who can delete a tag. The first implementation of a solution to this issue was as follows:

{
    “_id” : “MongoDB document ID” 
}, 
{  
    “albumId” : “numerical Apple Music album ID”,
    “tags” : [“An array of tags”, “stored as strings”],
    “authors” : [“An array of author names”, “stored as strings”] 
}

Here, a second array of authors was added. The indices in this array correspond to the indices in the tags array such that the tag at index 0 was created by the author at index 0. At this point only the user names were displayed, but that was sufficient to at least see who created a tag. To actually get this information on the page, I added Firebase’s Authentication API to the front-end of the site. I used this tool to allow users to log in to Album Tags with their google credentials. The benefit of a third party tool like this is that Firebase is actually the one who maintains the users API and PII, and I could limit the amount of actual user data in my database to simple information like display name or a Firebase auth token ID.

This approach worked great for data curation amongst a small group of users (i.e. I see Zach created this tag), but it did not restrict users from removing tags that weren’t theirs, and wasn’t very extensible should the site eventually be used by a wider group of individuals. The next step then was an upgrade to the user authentication process being completed on the front end of the site. I changed the “authors” array to store the userID token provided by Firebase instead of the display name that it started with. I then configured the front end script to display all the tags on the “Album Details” and “All Tags” pages, while only displaying tags from the authenticated user when on the “Update” page. This meant a user would only be able to delete their own tags, while still being able to preform tag searches using tags they and others collected.

As I mentioned earlier, one of my favorite parts of working on the Album Tags project was that the user side of my brain could make such specific feature-demands and really push the developer side of my brain to continually integrate new ideas and concepts. As I was learning Firebase to add authentication functionality, I was also playing with its Realtime Database API. The API allowed me to connect to a Firebase database directly from the front end of my application. The Realtime Database populates data over web sockets, which also means updates are very fast and easy to make.

I took advantage of the speed and flexibility of the Firebase Realtime Database to implement my next two site-wide features. For some background on the first feature, I’ll need to take you back to March of 2018 in the very first, experimental stages of Album Tags. The first version of the application was called “Music This Week,” and the site displayed a set of cards showing the top five albums I had been listening to that week. As the site expanded and became focused on tags, I found myself missing the curated-album-list experience of its earliest days. To bring this functionality back to Album Tags, I came up with the idea of a “user favorites” function. At first, this was just Favorites for me, still a part of the site on the “Our Favorites” page, but I quickly built out a second “My Favorites” page where the authenticated user’s favorites would be displayed. Users could use this functionality to add albums to their favorites list on the “Album Details” page (by clicking the heart icon.) At the start, favorites were modeled in the Realtime Database at an endpoint matching the users ID and as an array of Apple Music Album ID’s that user had favorited. (More on this later!)

The next functionality added using the Firebase Realtime Database was “connections”. These used a different database instance with a single endpoint and a Connection was modeled as an array with two Album ID’s. This meant that the connection would display on both sides as the opposite album, essentially making two updates with the one change. Though on the surface this feature might sound very similar to the way a Tag could connect two albums, it filled an important gap I was noticing in the Album Tags site. At times, I listen to an album and it immediately reminds me of another album. Sometimes I can identify a specific adjective or set of adjectives regarding genre, year, style, etc. that both albums share. In this case, tags can help connect these albums. At other times though, the albums sit in different genres but I still wanted to make sure they were directly connected in a way that would be easy to see. The new Connections feature allowed this by showing a new card on the “Album Details” page with images for each album that was directly connected to the album being viewed.

All these new features were great, except up until this point I was using a manual array of user emails or user ID’s to restrict who could make tag updates. There was also no way to distinguish who had added a connection. The application was super useful to me with all these features, but my friends were not able to access all the same functionality as I was unless I manually granted them access. Not only that, but I found myself wishing there was a way to only view my connections or tags, instead of only all the connections and tags. To complete both these features, I had to change the data structures again:

{
    “_id” : “MongoDB document ID” 
}, 
{
    “albumId” : “numerical Apple Music album ID”,
    “tags” : [“An array of tags”, “stored as strings”],
    “createdBy” : [
        {
            “tag” : “Tag string”,  
            “author” “User ID”  
        },  
        {
            “tag” : “Tag string”,  
            “author” “User ID”  
        }
    ] 
}

This new tag structure allowed me to reference the user who created each tag more directly when populating information about a single album, while still allowing easy access to all the tags associated with an album for tag-searching purposes. This was important because one of the big potential wrenches I was worried about was two users who wanted to add the exact same string as a tag. With this format, I could see who added each tag and allow each user to only delete their own tags without messing with tags created by any other users:

“Apple Album ID” : {  
    “albumId” : “Apple Album ID”,  
    “author” : “User ID” 
}

I made a change to the connections Realtime Database as well, to achieve similar user-specific functionality. A connection was now modeled as an album ID with the connected album ID along with the user who added the connection. The upgrade from a simple array meant that the code now had to make two updates with each addition so that each “side” of a connection could contain information about the other album.

With both of these data structure changes in place, I could now add functionality to the front end to allow users to toggle between all tags or connections and only their tags and connections. I could also finally remove the admin user restrictions I was manually implementing to allow any user who could sign in to Firebase with their Google credentials to add their own tags and connections. The site was officially public and any user could now take advantage of these new tools I had been getting such great milage out of!

This arrangement of data structures actually remained active for a few months while my friends and I continued to use the application every day. Along the way I encountered an unforeseen obstacle that I realized would not be resolvable with the current data structure. The obstacle that so rudely reared it’s head was that as Apple Music maintains its databases and bands move between music markets, Apple album ID’s can change. While building the application thus far, I had assumed these were permanent ID assignments. When this happens, there is simply no data returned when querying the old ID. One second it works, the next, completely gone. Unfortunately, my current data structures left very few clues for tracking down a new album ID to make an update in the database a change like this occurred on Apple’s side. To combat this, I added a third Realtime Database where a post could be made if an album ID failed, alerting me that intervention was required. I also added `albumName` and `artistName` as string attributes of an Album, a Connection, and a Favorite, changing the latter to an array of objects instead of just an array of integers.

This solution was able to help recover from a couple album ID changes and helped keep the site stable for the entire summer and fall of 2018. After some time though, and additional experience gained building other projects in different stacks and technology domains, I started to realize how cumbersome my Album Tags data arrangement had become. I had modeled a Connection, a Favorite, and an Album, and was using three or four separate databases plus many calls to the Apple Music API to populate some of my pages. It was great to have been able to add features so quickly as they came to mind over the previous months, but it was starting to look like time to take a new look at the big picture.

With this in mind, I went back to the drawing board. I re-examined the questions I was answering with my current data structure to think of other ways to solve the same problems. For example, when an album is favorited, a user needs to know this when updating the album, but users also have to be able to see all their favorited albums at once. One option I considered was creating a separate All-Favorites object to update alongside a single Album data model, but in the fourth iteration of the site I had noticed that the more API calls a feature required, the higher the chances of unintended behavior and data corruption on a slow connection (like a mobile device with poor service.) I held out for a data re-organization that would allow for a single Album data-model, and finally found a way to implement this while providing the functionality I had built for tags, favorites, and connections. Not only that, but I was able to do all this while simultaneously reducing the number of Apple API calls that would need to be made to populate a data-heavy page like the “My Favorites” page.

After several white boarding sessions working through each question that would need to be answered by the new data model, here is what the latest version of the Albums database looks like:

{  
    “appleAlbumID” : “String value for Apple Album ID”,
    “appleURL” : “URL to the listen in iTunes link”,
    “title” : “String value for the album title”,
    “artist” : “String value for the artist name”,
    “releaseDate” : “String date value for the album release”,
    “recordCompany” : “String value for the record company name”,
    “songNames” : [ “Array of strings”, “one for each song name” ],
    “cover” : “String url to the image for the album cover”,
    “genres” : [ “Array of strings”, “genres labeled by Apple Music” ],
    “tags” : [ “Array of strings”, “one for each tag” ],
    “tagObjects” : [  
        {
            “tag” : “Tag string”,  
            “author” “User ID”  
        },
        {
            “tag” : “Tag string”,  
            “author” “User ID”  
        }
    ],  
    connectionObjects : [
        {
            “databaseID” : “MongoDB _id value for the album being connected to”,  
            “appleAlbumID” : “Apple Album ID for the album being connected to”,
            “creator” : “User ID who created the connection”,
            “title” : “String album title”,
            “artist” : “String artist name”,
            “cover” : “String url to album cover image”  
        }  
    ],  
    favoritedBy : [ “Array of strings”, “each user ID who favorited this album” ]
}

This newest data structure solves quite a few problems. The entire site can now be built on a single Album model, which has tags, users that favorited it, and albums it is connected to. I also stored all the information needed to populate the Album Details page and the Favorites / My Favorites pages in the Album object itself so that no separate Apple API calls are required for albums already stored in the database. I was still able to pull all albums from the database that were favorited by a specific user, as well as all albums that match an array of tags being searched by the user. This meant that not only was the data structure simpler, but it was also more reliable and could serve as a strong foundation for future improvements as the site continues to grow in the year ahead.

In the most recent release of Album Tags, there were several new lessons learned and changes made. Here I focused on the improved data structure, but stay tuned here for lessons learned from the new back-end architecture and test driven design approaches as well!

Originally published at joshuahunschejones.wordpress.com on November 27, 2018.