T-Mobile, one of the world’s largest telecommunications companies, approached us to elevate their machine learning capacities. Specifically, they wanted to know whether their Continuous Integration and Continuous Delivery (CI/CD) stack for machine learning development was sufficient, especially compared with similar companies. We first had to identify topics of interest, and then we could benchmark the CI/CD stack. We combined our own knowledge, CI/CD development models and opinions from T-Mobile employees to come up with an actionable selection of topics: code repository, building, testing, deployment, notifications & monitoring, team & people, and miscellaneous.
Rating Machine Intelligence
Employees filled out a questionnaire based on initial topics, and from there, we held interviews to collect diverse assessments of the current situation and develop benchmarks against other companies. We suggested using pull requests after looking at the way code reviews were completed in the code repository. Another separate internal library repository was already used as a module for other projects to avoid copying all code to each repository.
We suggested using feature branches that merge back into the primary development branch. We gathered the necessary tools used for building a better code. We researched opportunities for test-driven development and created an automated test suite for the internal library. Even some generic model evaluation tests are adequate for the general internal library.
Collaborative Models and Prototypes
We used Docker containers to deploy models and orchestrated the process with Apache Airflow. A higher percentage of “first-time-right” deployment cycles was guaranteed by a tighter integration of these model deployments with other applications like the data warehouse. T-Mobile scores 4.5 out of 5 on team & people. People are trained and skilled to work with the relevant technology. We suggested improving the team by adding dedicated DevOps engineers. The best ranked teams have a close collaboration between data scientists, data engineers and DevOps teams.
Telecommunications Driven by Information
We combined all scores in a spider plot and added benchmarks from other companies for a side-by-side comparison. T-Mobile scored well specifically on building, deployment, and notifications & monitoring and teams & people. The biggest improvement came with testing, growing the team and adding DevOps skills and capacity. To evaluate proposals for upgrading the technology used in the CI/CD stack, we implemented Bitbucket instead of GitFlow to better integrate with the already existing Atlassian stack in place. We’ve analyzed the current CI/CD stack of T-Mobile and we’ve identified further areas for growth.
T-Mobile is now on their way to implement the suggestions to improve their CI/ID stack and employ the most cutting-edge data infrastructure to match the reach and size of their global telecommunications operations.