In the past years, software communities have invested enormous time and effort in creating libraries, frameworks, and tools that will help any development team materialize their idea. With all those building blocks, the developers can develop new powerful applications faster and safer.

All those available resources changed the traditional trend of building things in-house tailored to the specific need of the application into building components that stick to the community standards by composing all those building blocks and implementing only the missing pieces.

Furthermore, this new way of developing applications opened the door to a new challenge. In a fast-changing environment, the building blocks used in a project will change often and, from time to time, will contain breaking changes that require the development teams to adapt the application to the new requirements.

Without that time invested in updating all the building blocks, the product will be exposed to various threats and functional or performance issues.

To avoid such a scenario when any of its dependencies endanger the application, the organizations have started to invest in a dependency management program. With this program, the changes required to update all the dependencies will be appropriately planned and prioritized.

VMS companies and their technical challenges

Yonder is a technical partner for a large number of VMS companies. Being part of this unique market changed how we looked at various technologies and forced us to adopt an automation mindset. Furthermore, we must always look for out-of-the-box solutions that will allow our partners to materialize their ideas in a way that will last over the years.

Vertical Market Software (VMS) companies operating in critical sectors face unique challenges regarding dependency management. These challenges are magnified by the need to build software that endures for an extended period, often with the expectation that it will last indefinitely.

For Vertical Market Software companies, dependency management challenges are intertwined with the need for enduring, reliable, and secure software. Navigating these challenges requires a strategic and proactive approach, combining careful selection of dependencies, robust maintenance practices, and a commitment to staying abreast of evolving industry standards and technologies. By addressing these challenges head-on, VMS companies can build and sustain software solutions that stand the test of time in the most critical and demanding environments.

Long Lifecycle expectations

Software often has a significantly longer lifecycle compared to applications in other markets. This means the dependencies chosen at the project’s inception must be sustainable and maintainable over the long term.

The risk of dependencies becoming obsolete or unsupported is heightened, requiring meticulous planning to ensure ongoing support and updates for the software.

Regulatory compliance

VMS companies are typically subject to strict regulatory standards. They ensure compliance with these standards while managing dependencies, which poses a challenge, especially as regulations evolve.

Failure to comply with industry regulations can lead to legal consequences and compromise the integrity of the software. Dependency updates may be required to align with changing compliance requirements.

Security and vulnerability management

Security is paramount for VMS companies, and vulnerabilities in dependencies can pose severe risks. However, updating dependencies in long-lasting software without introducing new issues can be challenging.

The software may become susceptible to security threats if dependencies are not regularly updated. Balancing security updates with the need for stability becomes crucial.

Integration with legacy systems

VMS companies often have legacy systems with which the software must integrate. The dependencies must be compatible with these legacy systems, and managing integration can be complex. Failure to integrate seamlessly with existing systems can disrupt operations and compromise the efficiency of critical processes.

Vendor support and continuity

Relying on external vendors for specific dependencies introduces the risk of those vendors discontinuing support or going out of business. This challenge is exacerbated in the context of long-lasting software. Loss of vendor support can lead to challenges in maintaining and updating the software, potentially requiring a significant overhaul to replace discontinued dependencies.

Skill set preservation

As software lasts for an extended period, the skills and expertise needed to maintain it may become scarce or obsolete. New developers joining the team may lack familiarity with the chosen dependencies. The risk of knowledge gaps and challenges in onboarding new team members can increase, potentially leading to slower response times for addressing issues.

Technological evolution

Technology evolves rapidly, and dependencies that were cutting-edge at the start of a project may become outdated. Integrating modern technologies without disrupting the existing software poses a challenge. Failure to adapt to evolving technologies can make the software obsolete, hindering the organization’s ability to leverage advancements and meet changing user expectations.

Budget constraints

Ongoing maintenance and updates for long-lasting software can strain financial resources. Budget constraints may limit the ability to invest in the latest tools or to perform extensive refactoring. The software may become stagnant, lacking the enhancements and optimizations that could be achieved with more robust financial support.

User expectations

Users often expect uninterrupted and reliable performance. Introducing updates or changes, including those related to dependencies, must be carefully managed to meet user expectations. Disruptions or modifications perceived as risky may lead to user dissatisfaction or resistance, making managing updates a delicate balancing act.

This part identifies why dependency management is crucial for all software companies, specifically VMS companies. It should highlight why we should pay good attention to this and have it on our minds from the beginning. The next part of this article will be much more technical as we will dive into the Renovate Bot, a transformative tool designed to streamline the management of dependencies while automating the upkeep of your projects. We will examine the key elements that define Renovate Bot and its impact on the open-source community.

Renovate Bot in a nutshell

In the dynamic realm of software development, staying ahead requires more than just lines of code—it demands efficient management of dependencies and continuous innovation. Enter Renovate Bot, a transformative tool designed to streamline the management of dependencies while automating the upkeep of your projects. Let’s take a closer look at the key elements that define Renovate Bot and its impact on the open-source community.

At its core, Renovate Bot is a tool meticulously crafted to enhance dependency management, minimize vulnerabilities, and accelerate development workflows. Its mission? To empower developers by automating the arduous task of keeping dependencies up to date and ensuring that projects are leveraging the latest improvements and security patches seamlessly.

Renovate Bot introduces a set of key concepts that reshape the way dependency management is approached:

Managers: Renovate Bot employs “managers,” which are configuration files that define how dependencies within a repository should be updated. These managers act as personalized guides, ensuring that your project’s dependencies evolve consistently.
Presets: Presets serve as templates that encapsulate best practices for dependency updates across different programming languages and frameworks. Renovate Bot’s presets expedite the configuration process, allowing developers to dive into efficient dependency management with minimal friction.
Dependency Management: Renovate Bot doesn’t merely update dependencies—it ensures they are scrutinized for vulnerabilities, compatibility issues, and performance enhancements. It guarantees that your projects are fortified with the latest advancements.
Merge Requests Automation: The bot generates automated merge requests that include updates, allowing developers to review and integrate changes effortlessly. This automation not only speeds up the update process but also maintains a clean and consistent development pipeline.

Renovate Bot has garnered significant attention and admiration within the open-source community for its transformative capabilities. Its automated approach to dependency management has liberated developers from the drudgery of manual updates, enabling them to allocate more time to innovation and feature development.

Projects across various domains have embraced Renovate Bot, touting its ability to reduce maintenance overhead and enhance the overall quality of its software. The bot’s integration with platforms like GitHub has solidified its position as a valuable companion in the open-source landscape.

Dependency Management with Renovate Bot

Renovate Bot takes the manual effort out of dependency management. It automatically detects outdated dependencies, creates pull requests, and even applies the updates if configured to do so. This automation ensures that your project stays current with the latest releases, bug fixes, and security patches without constant manual intervention. Even if your project relies on npm packages for JavaScript, Maven for Java, or any other package manager, Renovate Bot understands the nuances of different ecosystems, making it a versatile solution for diverse software stacks.

Furthermore, it provides a highly configurable environment, allowing you to tailor its behavior to suit your project’s specific needs. From update frequency to version constraints, you have the flexibility to define rules that align with your development practices.

Getting started with Renovate Bot

The Renovate Bot offers very powerful documentation that can guide you step by step in making important decisions for your dependency management program. Furthermore, at every step, they will highlight the risks and possible side-effects of every decision or configuration.

In a nutshell, the Renovate Bot setup up will have four pillars:

Repository Integration: to unleash the power of Renovate Bot, begin by integrating it into your project repository. The process typically involves adding a configuration file (renovate.json or similar) to your repository. This file serves as the blueprint for how Renovate Bot should handle dependency updates.
Configuration Options: Renovate Bot offers an array of configuration options, allowing you to fine-tune its behavior. You can specify which package managers to target, set update intervals, define versioning constraints, and more. The configuration is designed to be human-readable and intuitive.
Review and Merge: Once configured, Renovate Bot begins its work. It diligently scans your project for outdated dependencies and creates pull requests accordingly. The pull requests are intelligently crafted, providing a summary of the changes and offering a seamless review process. Depending on your configuration, Renovate Bot can even merge these updates automatically after passing all tests.
Continuous Monitoring: Renovate Bot doesn’t stop once the initial setup is complete. It continuously monitors your project for new releases, ensuring that your dependencies remain current over time. This continuous monitoring is a proactive approach to staying ahead of security vulnerabilities and leveraging the latest features.

Our story

In our journey with Renovate Bot, we were inspired by others sharing their experience and how they tackle challenges. Because of that, we considered that it was our time to share what we managed to achieve using Renovate Bot with the community and how we overcame the challenges we encountered.

We would highly recommend having a look at the following user stories:

A small glimpse of the existing tooling

If you are already familiar with Renovate Bot, you already know that the community behind it is always improving the product in order to ease the adoption and create a smooth integration in existing ecosystems.

For us, the major challenge to overcome was to slightly change the way Renovate Bot is usually configured and used so we could integrate it with various technologies and tools that we already have in our landscape.

Before we start, we will try to give you a small overview of the tools that are part of the dependency management program in our ecosystem.

Repository templates (Cookiecutter and Cruft)
Cookiecutter is a template-based project scaffolding tool, while Cruft is a companion tool that helps maintain and update projects created with Cookiecutter. Together, they streamline project initialization and maintenance, ensuring consistency across projects and easing the burden of manual setup and updates. Together they enhance development efficiency by automating project setup and maintenance tasks. They promote consistency in project structures and configurations, reducing the risk of dependency-related issues.
GitLab CI templates
Because of the large number of components, we treat our GitLab CI pipeline definitions as a separate project. The project contains various fragments specialized for a particular task written in such a way as to facilitate the reusability and composition of various fragments.
As a result, each pipeline for a component is composed of various fragments, and the final results are dictated by the variables passed from each individual project.
Semantic release
Semantic Release automates versioning and package publishing based on the semantic versioning specification. It analyzes commits to determine the appropriate version number and generates release notes automatically. This tool facilitates consistent versioning and reduces the manual overhead of managing releases. It improves release management by automating versioning, making it easier to track changes, and ensuring that version numbers accurately reflect the significance of updates. This is especially valuable in projects with numerous dependencies.
Dependency track
OWASP Dependency Track is a platform for managing software component dependencies and tracking associated security vulnerabilities. It provides insights into the risk profile of a project’s dependencies, allowing teams to address security concerns proactively. It enhances security in dependency management by identifying and tracking vulnerabilities in third-party components. It enables organizations to make informed decisions about using and updating dependencies to mitigate security risks.
Defect Dojo
OWASP Defect Dojo is an open-source application security management tool designed for managing the entire application security testing lifecycle. It assists in tracking and managing security vulnerabilities, including those related to dependencies. It provides a comprehensive solution for managing security vulnerabilities, including those introduced by third-party dependencies. It aids in identifying, tracking, and mitigating security issues throughout the software development lifecycle.

Technology stack

Over time, the project and its infrastructure have undergone substantial growth and transformation, resulting in a diverse and extensive technology stack. Presently, the landscape encompasses:

100+ Custom Container Base Images – These serve as foundational building blocks for all components developed within our organization, providing a standardized base for containerized applications.

300+ Mirrored External Container Images – Community-maintained container images mirrored within our organization. This setup allows us to follow a distinct dependency patching schedule independent of the vendor’s release cycle.

300+ Container Images for Components and Tools – Specifically tailored container images for various components and tools integral to our ecosystem.

50+ Internal Tools – Custom tools are designed for unique workflows or sensitive operations where reliance on community tooling might be impractical. The top three programming languages employed in tool development are Python, Go, and Java.

100+ Java Components – Core components of the system developed using the Java programming language.

50+ C/C++ Components – Essential components are written in C/C++, contributing to the diverse language landscape within the project.

50+ Infrastructure-Related Projects – The infrastructure follows the Infrastructure as Code (IaC) principle, with all assets defined in code. This category includes projects that manage and orchestrate the underlying infrastructure.

200+ Legacy Components – Comprising components written in various older technologies, these legacy assets contribute to the project’s historical landscape and add complexity to the overall technology mix.

Requirements

To ensure a seamless and effective implementation of Renovate Bot within our diverse and dynamic ecosystem, we’ve established a set of key requirements that align with our project’s complexity and varied component types.

Versioned configurations
All configurations must be versioned to maintain control over the evolving ecosystem. Pinned versions are a cornerstone, allowing precise management of promoting mechanisms based on team requirements.
Custom presets for component types
Given the diverse technologies present in our ecosystem, custom presets tailored to each component type are essential. These presets accommodate distinct requirements and limitations associated with different technologies.
Modular configurations to avoid duplication
To promote maintainability and reduce redundancy, configurations should be modular. Common aspects are encapsulated in reusable fragments, while component-specific presets extend or modify the generic presets. This approach minimizes duplicated configurations.
Automatic onboarding
Onboarding should be automatic, leveraging Renovate Bot’s ability to auto-discover repositories. The process should seamlessly identify repositories and provide dedicated configurations, streamlining the integration of Renovate Bot across the organization.
Selective merging for sensitive changes
Renovate Bot should offer the flexibility for teams to decide what changes are merged, especially for sensitive modifications. This ensures that teams retain control over critical aspects, aligning with security and compliance requirements.
Minimizing noise in the organization
A key objective is to keep organizational noise to a minimum. Renovate Bot should facilitate the promotion and validation of small changes under the hood, ensuring that routine updates do not disrupt the day-to-day operations while maintaining the health of the ecosystem.

The implementation journey

Repository classification

The first milestone was to add additional topics/labels on each repository in order to allow us to group repositories by their type later on.

An example of the topics we used and their possible values can be found below:

Renovate Bot self-hosted presets

Even if the Renovate Bot community offers an impressive collection of presets, we decided to rely only on self-hosted configurations. Implementing self-hosted presets for Renovate Bot offers several notable advantages, providing a tailored and controlled approach to dependency management within your organization.

The self-hosted configuration files are stored in a dedicated repository named “renovate-config” and are grouped by the following categories:

default: the generic configuration file that contains rules that every repository from the organization should follow.
auto-merge-*: rules for allowing particular changes to be merged automatically by the Renovate Bot.
component-*: the final configuration file for a group of repositories from our organization. Each component is a collection of other configuration files included in the “renovate-config” repository.
group-*: presets that will group together multiple dependencies to reduce the noise.
preset-*: reusable configurations that address a certain particularity from our infrastructure.
replacement-*: presets that will try to replace older components or will tackle the components that should be migrated.
schedule-*: various configurations regarding the patch management schedule for each type of component
team-*: team-specific configurations that should applied on certain components.

Table1: The “renovate-config” repository structure.

Some of the key aspects that influenced our decision were:

Granular control of changes
Self-hosted presets allow for precise control over the configurations used by Renovate Bot. Teams can curate presets to align with specific project requirements, ensuring that only approved changes are applied to the codebase. This granular control enhances the predictability and stability of the development environment.
Tailored to project requirements
Each project within our organization may have unique dependencies, versioning strategies, and release cycles. Self-hosted presets enable customization based on project-specific requirements. This tailored approach ensures that Renovate Bot aligns seamlessly with the diverse needs of different teams and projects.
Centralized management
Hosting presets internally provides a centralized location for configuration management. This centralization streamlines the process of updating, versioning, and maintaining configurations. Teams can efficiently manage and share presets across projects, promoting consistency and standardization.
Versioning individual configurations
With self-hosted presets, individual configurations can be versioned independently. This capability ensures that changes made to specific configurations can be tracked, rolled back if necessary, and audited over time. Versioning enhances accountability and facilitates a more controlled evolution of configurations.
Customization for diverse technologies
In organizations with a diverse technology stack, self-hosted presets accommodate different languages, frameworks, and component types. This flexibility allows teams to tailor configurations based on the specific needs and nuances of various projects.
Security and compliance requirements
Hosting presets internally aligns with security and compliance considerations. It enables organizations to implement and enforce configuration standards that adhere to internal policies, industry regulations, and security best practices.

Useful resources:

Renovate Bot setup

We decided to keep all the configurations for the Renovate Bot Runner and dedicated onboarding configurations in a dedicated repository named “renovate-runner.” The renovate runner will have three major responsibilities:

to discover all the repositories that should be processed during the execution.
to onboard all the repositories that match the filters (if necessary)
to apply all the required changes to the repositories discovered in the previous step.

The common approach is to have a single instance of Renovate Bot running for all the repositories that should be processed. Because we wanted the repositories to be automatically onboarded in the dedicated configuration file, we decided to split the execution of the Renovate Bot based on the repository classification.

By doing so, the repositories will be automatically enrolled with the latest configuration available based on their type. The downside of this approach is that we will always have multiple instances of Renovate Bot running at the same time.

Lucky enough for us this downside was desired because now the Renovate Bot will manage to get through all the repositories faster without causing any harm to our VCS and its job runners.

Table 2: The “renovate-runner” repository structure.

Each onboarding configuration file will contain component-specific configurations and a common part that will allow Renovate Bot to select the desired repositories:

{repository-type}: the type of repositories that should be handled by the current instance of Renovate Bot
{repository-path}: the location of the “renovate-config” repository.
{component-type}: the type of the current component used for selecting the right Renovate Preset
{renovate-config-version}: the latest version approved for the “renovate-config” repository.

Useful resources:

Putting all things together

After preparing the “renovate-config” and “renovate-runner” repositories, the only remaining task is to let the Renovate Bot do its magic. For us, the missing glue was some GitLab CI pipelines that will execute the Renovate Bot CLI.

At the first run, Renovate Bot will discover all the repositories that have the “renovate-bot:enabled” topic set and will create an onboarding Merge Request containing the component-specific configuration. Each project maintainer needs to review the Merge Request and all the additional information included by the Renovate Bot in the Merge Request description.

Once that Merge Request is merged, the team can stand back, relax, and let the Renovate Bot find all the dependencies that should be upgraded.

Alex Coman
Software Architect

Did you miss the previous articles by Alex on Dependency Management? Then please click on the articles below. If you have a question for Alex, please get in touch with Alex directly.

Dependency management strategy.

Best practices for dependency management.

How to renovate hundreds of repositories overnight

The third article in the series on dependency management