Aim
There is more pressure than ever for developers to deliver faster and be more agile without sacrificing security, quality, or reliability. A strong DevOps practice aligns the software development and operations teams to enable faster code deployment, automated testing, and continuous feedback loops, which allow for iteration, more collaboration, and better business agility.
Cultivating a DevOps culture was required to support UCSF’s fast-moving research and patient care standards. DevOps provides rapid, consistent delivery and the ability to operate with less downtime and fewer security issues, all of which are necessary for innovation at UCSF. Automated testing in particular saves time and improves accuracy during the quality assurance (QA) process, which otherwise demands considerable manual effort for complex operations.
In June 2018, UCSF’s Center for Digital Health Innovation (CDHI) launched a custom referrals automation application, which converts patient referrals into electronic medical (EMR) referrals. The application requires integration with multiple systems across UCSF, commercial vendors, and other third-party systems. CDHI wanted to implement a DevOps practice to automate the build, test, and deployment processes as much as possible in order to iterate quickly and still maintain stability.
“I knew that SOM Tech shared our vision of a DevOps culture and had the right experience and resources to partner with CDHI to expand our DevOps practices. We had been using Agile methodologies already, and we moved to Amazon Web Services to take advantage of programmable infrastructure,” said Ed Martin, CDHI Technology Director. “It was critical for long-term sustainability that we develop these additional DevOps competencies.”
Approach
The project team used an agile approach to identify and implement the tools necessary to create a responsive DevOps culture.
We began by selecting an automated testing tool, which we did by identifying what criteria we were testing and what environments we were testing in. While we had a strong DevOps understanding, we still faced significant constraints and challenges, including:
- APeX integration (UCSF’s Epic EMR system)
- External API integration (Redox, Interconnect, NIH, and other third-party APIs)
- Working across multiple environments (development, QA, production)
We identified three automated test cases containing the key aspects of the platform to be tested: user interface (UI), application programming interface (API), and functional testing. Using these test cases, we performed a market scan of automated testing tools, did a deep dive analysis on the tools that most matched our criteria, and identified one tool (LeanFT) that was strongest in meeting all UI and API needs.
Another critical component was to define the flow and build out the continuous integration/continuous deployment (CI/CD) tools. CI/CD is part of a DevOps software development practice that allows teams to develop, test, and release software quickly and consistently through an automated process each time code changes. CI/CD requires not only the right automation tool but also code management (GitHub) and a robust delivery system that is compatible for almost any combination of languages and source code repositories (Jenkins).
This was our process flow at the highest level:
- Developer checks code into the GitHub repository
- Jenkins retrieves and compiles the code
- Jenkins deploys the code to the shared development or test environment
- A successful deployment to test triggers Jenkins to run the automated testing scripts via LeanFT
- Each step in the process includes feedback loops via Slack and email
The team encountered several anticipated hurdles while building out the process flow, such as integrating with the UCSF network while maintaining security. Epic integrations proved challenging as well and required strong partnership with UCSF IT to ensure servers inside and outside the firewall were connected. We relied on our agile project methodology, with its focus on collaboration and communication, to overcome these challenges and keep moving forward.
Solutions
We completed CDHI’s DevOps platform in March 2019. The platform automates the referral application’s entire build and deploy process as well as three automated testing flows. The workflow creates a predictable, continuous process in which code is securely deployed and built-in feedback loops through email and Slack ensure problems are immediately identified.
This solution has increased testing quality and coverage, reduced the need for manual testing, and reduced the time it takes for code (e.g. new features, fixes, etc.) to reach users. On average, it takes:
- 45 seconds for Jenkins to retrieve developers’ code from GitHub
- 37 seconds for Jenkins to compile their files
- 4-6 seconds to deploy the compiled code to the appropriate environment
CDHI DevOps Workflow
