Skip to content
Fundament Docs

FUN-11 Plugins and the Marketplace

Pre-Discussion

1. Introduction

Fundament takes a minimalistic approach when it comes to Clusters, which are lightweight by default and have only the bare minimum installed. To add functionality, users may either decide to roll their own platform services and applicatios, or rely on the experience of others to provide packaged solutions. Plugins are a way to install and share solutions ranging from basic platform needs (object storage, gateway, dns) to complete application suites (zaaksysteem).

2. Functional requirements

  • Fundament must provide users with a generic system for installing/uninstalling of service/platform functionality into their Clusters. For now, we call this 'Plugins'.

  • Fundament must provide users with a single-pane-of-glass of all services and tools provided by Plugins.

    • Organization admins need to be able to configure and observe Plugins installed in Clusters.

    • Project members need to be able to see resources that are deployed (managed by the Plugin) in their Projects.

      • For example: cert-manager plugin needs to show users the CertificateRequests that are being tracked in their projects.

  • Plugins must be installable in Clusters that are airgapped/offline. It is up to the implementation of the Plugin whether it is itself compatible in an airgapped/offline environment.

  • Fundament must own the complete failure domain of the Plugin installation lifecycle.

  • The plugin system by Fundament must be initially simple; Plugins themselves can become more complex when required.

  • Plugin implementations must have the flexibility to decide themselves whether they are configurable through their own CRD or by exposing the CRDs of a tool/application they may be wrapping.

  • Plugins may declare (optional) dependencies, but this is not enforced by Fundament. The plugin must be built to be eventually-consistent.

  • Plugins must declare their requirements upfront (e.g.: when a plugin needs internet access, it must declare this so the user can approve the requirements before a plugin is installed)

3. Technical requirements / vision

A Plugin Definition declares the outline of the Plugin, can look something like this:

---
version: v1.0.0

requirements: internetAccess: true kubeSystemDeployment: false # When true, only possible to install this plugin on a warranty-voided/rooted Cluster. privileged: false # When true, only possible to install this plugin on a warranty-voided/rooted Cluster. capabilities: # ?? - storageClass # ?? - loadBalancer # ??

dependencies: - storage

suggestedPlugins: - observability

menu: cluster: - crd: ClusterIssuers list: true detail: true edit: true project: - crd: Issues list: true detail: true edit: true - crd: Cerficicates list: true detail: true edit: false

runtime: image: ghcr.io/foo/bar:v1.0.0@sha3-blablabla ---

A Plugin is installed into a Cluster by creating a Namespace and Deployment. The Deployment runs the Plugin Runtime container image. The Deployment manifest is generated by Fundament based on the values from the Plugin Definition, this can be performed by the Fundament plugin-worker (nephew of the cluster-worker).

When a Plugin is installed in a Cluster, the Console UI will display a new menu-item on both the Cluster page and the Project page (for all Projects that are deployed on the Cluster where the Plugin is installed). The menu-item may have sub-menu items, this is decided by the Plugin Runtime console-ui-plugin/AngularPlugin/tbd. For example: when cert-manager is installed in a Cluster it will show:

  • In the Cluster menu (for the organization admin) the CertificateIssuers

  • In the Project menu (for the project member) the Certificates and CertificateRequests.

These are only one and two resources respectively, but for a consistent experience it still makes sense that every top-item in the menu is the Plugin name itself. This provides a consistent experience for the users.

The Plugin Runtime has a number of functions:

  • Installation / Reconciliation of applications.

  • Metadata API for Fundament

    • List the current state of the plugin (the Runtime is available, but it may still be installing)

    • List Roles that can be assigned to users

  • User-interface (¿AngularPlugin?) for the configuration of the plugin (A User who is organization admin views "Cluster > Plugins" in the console)

  • User-interface (¿AngularPlugin?) for the resources the plugin manages (A User who is project member views "Projects > CertManager > CertificateRequests" in the console.)

Although the Plugin Runtime hosts the interface components for console-ui, it does NOT proxy data towards that interface. This is done through normal kube API and follows the k8s RBAC.

The Plugin Runtime must be upgradable within the same namespace as long as it’s the same major version of the plugin. For example, it will not be possible to install storage at v1.4.0 and later upgrade it to v2.0.0. If a plugin wraps a helm chart, it may make sense to version the plugin at the same version as the helm chart.

Scenario:

  1. Organization Admin installs Plugin "cert-manager" into Cluster "foo" at version v1.19.3.

  2. Fundament plugin-worker creates a namespace plugin-certmanager-v1.

  3. Fundament plugin-worker creates a deployment plugin-certmanager-v1/runtime with image forgejo.digilab.network/fundament-plugins/cert-manager-v1:v1.19.3@<sha hash>.

  4. The Plugin Runtime uses helm to install cert-manager in its own namespace.

  5. The Plugin Runtime starts hosting metadata server. (Fundament’s plugin-worker connects to the metadata server and pulls the plugin status and available Roles (Viewer, Editor)).

  6. The Plugin Runtime starts hosting console-interface, which is loaded by console.fundament.localhost (implementation details tbd)

  7. An organization admin assigns 'CertManager Editor' Role to users Frits and Joost in {cluster=foo, project=bar, namespace=baz}

  8. Frits navigates to the console-ui and opens the Certificates page.

  9. The Certificates page loads UI components from the Plugin Runtime in some magical way that we still have to decide on.

  10. The Certificates UI component needs data to display, it uses the (injected) kubernetes client to list Certificates. (Ideally the injected kubernetes client has an attached SA with 'dropped' permissions; to only the roles of the plugin itself. Such that one plugin cannot interact with resources of another plugin).

  11. Since Frits has the 'CertManager Editor' Role in namespace baz, he is allowed access by kube-api to view Certificates in that namspace. The Certificates from namespace baz are displayed in the console UI.

  12. Joost is not a big fan of visual interfaces and uses terraform to modify the Certificate manifests; uses a standard kuberenetes terraform provider for that.

  13. Hannes tries to also modify a Certificate via kubectl, which would’ve worked fine if Hannes had been granted the Editor Role. Hannes gets an error from kube-api.

4. Marketplaces

Plugin Definitions can be provided by a marketplace. We can start with just a go:embed of a directory with Plugin Defintion files, but eventually need to look at a system that can be externally hosted with a fundament backend worker can sync on regular intervals. The Plugin Definitions must be properly signed.

Plugin Definitions can be sideloaded; manually inserted into Fundament by an Organization Admin. The sideloaded Plugin Definition will only be visible within the Organization that it was sideloaded to and can only be installed in Clusters within that Organization.

For airgapped/offline clusters, plugins must be able to host containerimage on the Fundament servers (registry). This means a plugin becomes more than just a Plugin Definition file. v0.3?