Many organisations embark on complex software systems supporting critical business processes involving humans to deliver value. For instance, they may deploy a large data platform that is shared between various business divisions within the organisations and/or with other organisations. While there is often a lot of money invested when creating the software before the first go-live, there is often little governance involved during the evolutive phase of a complex system: How can the system evolve taking into the account all stakeholder views and creating a common understanding on complex business/technical aspects? How can we design evolution of one system to a completely different system without creating a project redesigning everything from scratch?
I will investigate in this post so-called improvement proposals, such as Python Enhancement Proposals (PEP), used in large scale open source projects to derive implications for their use in enterprise settings. Unfortunately the scientific research provides only few insights on the topic (cf. here or here).
While many projects have implicitly a process through issue management to track these changes, I will focus here on large scale open source projects involving contributors from multiple organisations that have laid out and documented the process. In future blog posts, I may extend the investigation towards also other projects.
The next section analyses various improvement proposals in open source projects including guidelines to create them. From there I will derive a common set of criteria in open source projects. Afterwards I will discuss challenges in changing complex software systems that affect multiple business divisions or other organisations by using the example of a shared data platform. Finally, I will discuss how learnings from improvement proposals can be applied to complex enterprise software systems.
Existing Improvement Proposals in the Open Source World
The following non-exhaustive list provides an overview on improvement proposals in large scale distributed open source projects.
| Name | Scope | Year established | Guideline |
| Java Community Process (JCP) – Java Specification Requests (JSR) | Specification for Java Technologies | 1998 | https://www.jcp.org/en/procedures/overview |
| JDK Enhancement Proposals (JEP) | Java Development Kit (JDK) | 2011 | https://openjdk.org/jeps/1 |
| Kafka Improvement Proposals (KIP) | Major Changes to Kafka | 2015 | https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals |
| Linux Kernel Development Process | Patches to the Linux kernel | 1991 (note: the process emerged over time, but still Linus Torvals is in control) | https://www.kernel.org/doc/html/latest/process/development-process.html |
| Kubernetes Enhancement Proposal (KEP) | Changes to Kubernetes | 2016 | https://github.com/kubernetes/enhancements/tree/master/keps/sig-architecture/0000-kep-process |
| Python Enhancement Proposal (PEP) | New features, processes or environment | 2000 | https://peps.python.org/pep-0001/ |
| Internet Engineering Task Force (IETF) – Requests for Comments (RFC) | Internet technical foundations | 1969 | https://www.ietf.org/process/rfcs/ |
| Rust – RFCs | Changes to Rust | 2014 | https://github.com/rust-lang/rfcs |
| Solr Improvement Proposal (SIP) | Major Enhancements to Solr | 2019 | https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=135862979 |
| Spark Project Improvement Proposal (SPIP) | Major improvements in the Spark Codebase | 2016 | https://spark.apache.org/improvement-proposals.html |
One can observe that the mechanism existed as early as 1969, but only since 2015 (with some exceptions) it gained traction. It could be that large projects involving multiple stakeholders became more frequent in the 2010s.
We also characterized here the Internet as an open source project, but the Internet is not just one project and the IETF mainly publishes standards/protocols. Nevertheless, the architecture foundation of the Internet is based largely on open source.
Another observation is that improvement proposals exist for very mature projects and they usually appear within few years after the open source project is created. They also seem to ensure stability and trust into an open source project.
Types of Improvement Proposals
One can distinguish between different types of improvement proposals:
| Type | Example |
| Changes to the architecture and design of an open source project (often includes an implementation) | KIP-932: Queues for Kafka SIP-12: Incremental Backup and Restore JEP 444: Virtual Threads |
| Changes to the governance structure of an open source project | JEP draft: JEP 2.0, draft 2 PEP 1 – PEP Purpose and Guidelines |
| Changes to protocols and standards | RFC 7504 – SMTP 521 and 556 Reply Codes |
| Good practices around using the technologies provided by an open source project | PEP 672 – Unicode-related Security Considerations for Python RFC 8725 – JSON Web Token Best Current Practices |
| Research Proposals | RFC 9631 – The IPv6 Compact Routing Header (CRH) |
| Humorous: Some projects have few proposals of humorous nature. This is particularly interesting as they help to define borders between senseless proposals and proposals useful for the community around an open source project | RFC 9564 – Faster Than Light Speed Protocol (FLIP) |
There is different type of formality to an improvement proposal. Some are more formal according to a template and some are simply sending code/patches.
Often proposals that affect architecture/design are complemented by code demonstrating the advantages, for example, in terms of performance or security.
Interestingly the software lifecycle also come into play. While most are at the „beginning“ of a software lifecycle, ie design/implementation there are also some on operation (e.g. best practices) or retirement of outdated features/dependencies.
There are no clear cut criteria when an improvement proposal should be raised or when a simple bug or feature enhancement opened (e.g. the JEP states different criteria of which one must be selected, such as minimal 2 weeks engineering effort, high demand or significant changes). Clearly, due to the more heavy processes around an improvement proposal, there tend to be much less compared to regular issues.
Benefits of Improvement Proposals in the Open Source World
There can be many reasons why improvement proposals exist, such as
- Complexity of the problem to solve does not justify raising it as an issue, but requiring research and possibly expert input from inside/outside the project
- Possibly many stakeholders in the project (e.g. users, developers, commercial actors) affected or complex project governance structure
- The solution is not obvious for an expert in the field or no one has done it before
- Solution described in an improvement proposal cannot be provided in a sufficient quality and requires time to mature
- Quality assurance (e.g. performance, security) is a significant effort
Those goals are not often stated explicitly.
Improvement proposals are NOT:
- It is not a feature request
- It is not a bug report
- It is not an idea/innovation tool
These things do exist in open source projects that have additionally also improvement proposals. As stated before improvement proposals bring complex changes to software and the business processes implemented in them. They have some risks, but the goals should be clear and how to achieve them.
Lifecycle of an Improvement Proposal in the Open Source World
Different projects have different lifecycles for improvement proposals. Additionally, they have different preconditions for entering a lifecycle state. Furthermore, there are differences between the „official“ documented lifecycle and the „lived“ lifecycle of improvement proposals (e.g. see here). They also evolve over time. We consider here only the „official“ documented ones as determining the „lived“ lifecycle is an enormous effort to analyse. They can though be very different. Naturally different actors in the lifecycle may have different power within the phases and the whole lifecycle.
Reasons for introducing informal lifecycle phases are often a changed environment, project maturity and/or governance.
Find here a non-exhaustive list of lifecycle phases from official improvement proposals. The first column introduces a proposed generic name and the second column how it is named in different improvement proposals mentioned before. The third column provides a description and the fourth column, precondition, any requirements before entering the phase. Not all improvement proposals have the same set of preconditions. Thus, the list is indicative.
| Lifecycle Phase | Corresponding *IP | Description | Precondition |
| Propose | Draft Development (JCP) Community and Public Preview (JCP) Draft (JEP) Posted (JEP) Submitted (JEP) Provisional (KEP) Early review (kernel) Draft (PEP) Informational (RFC) Experimental (RFC) Proposed Standards (RFC) Pull Request (Rust RFC) Build Consensus and Integrate Feedback (Rust RFC) | Experts from the community write an improvement proposal. The draft is usually public or accessible to community members | Approval from community to write proposal, community may designate experts |
| Vote | Approval Ballot (JCP) Candidate (JEP) Implementable (KEP) Wider review (kernel) Provisional (PEP) Accepted (PEP) Publish (RFC) Motion for final comment period (Rust RFC) | Vote on accepting the improvement proposal by the community or a dedicated community representative group/person | Often several review rounds by designated leads |
| Implemented | Final Release (JCP) Funded (JEP) Completed (JEP) Implemented (KEP) Merging into mainline (kernel) Final (PEP) Internet Standard (RFC) Best Current Practice (RFC) Merge (Rust RFC) | Make improvement proposal and a possible reference implementation available | Successful vote for an improvement proposal |
| Live | Maintenance (JCP) Active (JEP) Replaced (KEP) Implemented (KEP) Stable Release (kernel) Long-term maintenance (kernel) Active (PEP) | Manage changes to an improvement proposal or decide on superseding it by another successful improvement proposal | Optional: Public comments taken into account from implemented phase |
| Exception | Withdrawn (JEP, KEP, PEP) Rejected (JEP, KEP, PEP) Deferred (KEP, PEP) Historic (RFC) Close (Rust RFC) Postpone (Rust RFC) | Improvement Proposal is either withdrawn by an author or rejected by the community to not worthy to continue to pursue |
Not all open-source projects have an explicit governance process, e.g. the Go or Kotlin have a much less transparent process.
It can take different times until a proposal is gone through the process.
Governance
There can be different roles involved in an improvement proposal and its lifecycle. Those roles might be fulfilled by one or multiple persons. A person may have several roles.
| Role | Description | Persons involved |
| Proposer | Proposes a new improvement proposal | Community member (JCP) Individual Committers (JEP) Anyone (Linux Kernel, KIP, KEP, PEP, Rust RFC, SIP, SPIP) RFC Editor (IETF RFC) |
| Implementer | Implements an improvement proposal (if it requires implementation) – often the same as proposer | Often the same as proposer |
| Reviewer | Reviewer of the improvement proposal | Anyone/Community (JCP,Linux Kernel,KIP,PEP,IETF RFC, SIP, SPIP) Individual Committers (JEP) „Chosen“ (KEP) Specific subteams in community (Rust RFC) |
| Voters | Accept improvement proposal | Executive Committee (JCP) Group/Area Lead (JEP) Anyone/Community (KIP, IETF RFC, SIP, SPIP) Subsystem Maintainer (Kernel) Special Interest Groups (KEP) Steering Council (PEP) Specific subteams in community (Rust RFC) |
| Shepherd | Ensures smooth and timely transition of a proposal through the lifecycle phases | Editor (KEP) Champion (PEP) Shepherd (SPIP) |
Another interesting aspect is how those people get the roles they have in an improvement proposal.
| Role | Selected How |
| Proposer | Anyone (possible restricted to official community membership) |
| Implementer | Often the same as proposer / the same conditions as proposer |
| Reviewer | Often the same as proposer / the same conditions as proposer |
| Voter | Benevolent dictator for life (BDFL) Recognition by the community for reliability and quality (e.g. Linux (subsystem) maintainer) Election (e.g. JCP) Nomination Committee (e.g. IETF NomCom) All community members can vote |
| Shepherd | Based on previous community contribution Volunteering (and no objection by the community) |
Sometimes I refer here to community member. However, this is not always an open community that anyone can join. For example, JCP Community members need to register (which is nowadays free and in theory open to anyone). The more decision/voting power on improvement proposals exist, the more exclusive several of the open source projects become (e.g. JCP Executive Committee or Linux kernel subsystem maintainer). This can be based on contributions to the community, but is not the only way (e.g. sometimes it is based on power of individual companies within the community).
Tools
The previous section showed that an improvement proposal can go through different „official“ (publicly described) lifecycle phases, but also non-official lifecycle phases. This requires a flexible tooling to deal with this process.
| Tool | Found in |
| Linux Kernel mailing lists | |
| (Custom) Editor tool | IETF Datatracker |
| Source Code Versioning (e.g. as markdown files) | KEP, PEP, Rust RFCs |
| Issue Tools | SIP, SPIP |
| Wiki | KIP Wiki, SIP Wiki |
It can be observed that the tools allow a high degree of flexibility, which seems obvious since an improvement proposal entails some uncertainty/risk. Additionally, a combination of multiple tools might be used (e.g. a task management tool, an external web-based rich text editor and an open diagram tool). Others, e.g. PEPs use a discussion forum and a source code versioning tool.
Complex Enterprise Systems and Challenges to Improvement
Improvement proposals can also be very useful for internal and external (customer-focusing) enterprise systems. However, there is barely any literature or research found on the web on this. Most often the concept of an improvement proposal is not used and I argue here that this is wrong, because a joint improvement proposal process can reduce risk and deliver higher value as it can include a much broader and diverse perspective of all enterprise stakeholders.
I include here also enterprise systems that spawn the boundaries of multiple enterprises (e.g. supply chains), because for them having a clean and open improvement process for inter-organizational systems is even more important as there is a higher diversity of organizations which have different goals.
Enterprise systems face very similar challenges as open source projects. They can be in fact even more complex as there are internal users (e.g. employees) and external users (e.g. customers, suppliers) within complex business processes requiring complex business logic implemented in software. Often these users can contribute significantly to reduction of risks and realization of benefits. An enterprise system can be core to its business (* even for small companies) and failure to evolve it may mean significant losses. Many divisions need to work with an enterprise system to achieve a common goal (e.g. Enterprise Resource Planning (ERP) systems), but changing one part relevant to them may impact others.
Other technical aspects have significant impact on risks and value, such as IT Security, Operations or environmental sustainable systems.
Given the lack of research and knowledge one can only assume how improvement proposals for enterprise systems should look like. Over time and with experience this will change.
What could be possible differences to improvement proposals in complex open source projects? One may argue that enterprise systems are less technical. While this may often be the case, I think it is just a different level: Usually enterprise systems have complex business logic representing complex business processes and thus having a large community of users with expertise in it can be beneficial. Others may say that enterprise systems have much less users than complex open source projects. This depends: Some might have less users, but in larger enterprises or when multiple enterprises are involved they can have an equal large user base. Nevertheless, the size of the user base does not matter – improvement proposals aim at reducing risks and increasing benefits – and here even few users that can contribute matter.
How could one start improvement proposals for enterprise systems? I suggest – until we know better – to start similar as for improvement proposals for open source projects:
- Identify the community of an enterprise system: This can be users, developers, customers, sponsors etc.
- Have an open transparent process for improvement proposals within the community. You do not need to necessarily publish it to the world, but to the community and this includes, but is not limited to, designs, code etc.
- Use right tools for publishing: Your Enterprise should have an open wiki and/or a source code versioning system and/or an issue tracker. The exact choice of tools depends on your community. Use already well-established tools. No user will just use a tool for improvement proposals.
- Make clear who decides on moving improvement proposals forward. That is usually a person that has a budget. It can be also a chosen person that has contributed in the past significantly to the enterprise system (e.g. distinguished engineer) and budget and people is given to that person.
- Have shepherds that move improvement proposals further in the enterprise and that connect the right people to the improvement proposals
- Focus on improvement proposals which bring complex features/changes to enterprise systems. These are not innovation projects. The improvement proposal must clearly outline the benefits and how the change should be done.
- Your first improvement proposal should outline governance and process for improvement proposals
I think this is a good start until we have researched and understood better how the improvement proposal process for large scale enterprise system will work. Maybe in the future enterprise systems and large scale open source systems will converge to the same process as everything will be open?
Conclusions
This blog post is just a start to think about improvement proposals for large scale enterprise systems. It is a valid approach that has demonstrates its success in massive large scale open source projects. It is plausible to apply it to enterprise systems, because we find often that projects – let them be agile or waterfall – often not bring all the necessary people on-board, have issues integrating enterprise-wide perspectives (although they often affect enterprise-wide), do not support managed evolution of enterprise systems according to strategic needs.
You should definitely try it for your (inter-organizational) enterprise system. One may not get it right in the first attempt, but this is normal. The core principle is that it should be open to anyone in the community of your enterprise system (really anyone!). It should not be an idea tool, not a project charter and not a feature request or bug report: Improvement proposals describe complex changes requiring multi-perspective input and prioritization. They have a core technical focus realizing business benefits.
There are now multiple ways to move the topic improvement proposals itself forward:
- Research and understand the role of improvement proposals in the open source world deeper. There is a scarcity of research focusing only on very few projects. However, all the material is open – we just need to collect an analyze it. Example questions to be answered:
- How many proposals do exist per proposal type?
- How many proposals affect which part of the software lifecycle?
- It can be observed that improvement proposals were the key success for decades-old extremely complex projects (e.g. Linux or the Internet). What are success factors?
- What separates improvement proposals from other software processes, such as feature requests, bug reports, projects etc., aside of being complex requiring multi-perspective inputs?
- Are the tools sufficient to support improvement proposals? What is missing?
- How do improvement proposals support constructive evolution of software?
- Research and understand the role of improvement proposals in large-scale (inter-organizational) enterprise systems. Here the situation is more difficult to analyse. Despite the significant benefits for large-scale open source systems, we know very little if enterprises have similar approaches for their internal systems. Example questions to be answered:
- What are the benefits realized within an enterprise or cross-enterprises? E.g. better integration of regional views of an enterprise? Better understanding of all stakeholders?
- What already established tools in enterprises can be used for the improvement proposals?
- What are issues in integrating a multi-perspective view on improvement proposals in an enterprise or cross-enterprises?
- How to keep the community alive?
- How to identify and promote key contributors to a community?
Schreibe einen Kommentar