== Overview

The API server has been used to provide AddressSpace, Address and MessagingUser resources with the
potential of backing these resources by a separate persistence store.

However, the rest of the Kubernetes community is focusing mainly on CRDs, and CRD capabilities are
improving for every release.

Using CRDs will provide the following benefits:

* The API server can be removed - one less component to maintain and operate
* API features like watch, field selectors and more are will 'just work' without any implementation effort on our side
* Resource <-> ConfigMap conversion is no longer needed. Controllers can use the correct resource type

This proposal provides a path for migrating from the current API server based resources to CRDs.

== Design

The API server also functions as a validation mechanism, as well as for attaching resource labels and annotations that are used by controllers. With CRDs, validation is restricted to OpenAPI V3 validation. Additional validation have to be implemented as resource status conditions and optionally as validation webhooks. Labels and annotations have to be added by controllers in the first instance.

=== AddressSpace

The `AddressSpace` resource is used by the enmasse-controller-manager, address-space-controller and
service-broker components. These will have to change to watch and operate on the CRD resources directly
rather than using the ConfigMap-based implementation. Libraries already exist for that, so the amount of change is small.

The following validations are performed on an `AddressSpace` and will have to be moved to an OpenAPIv3 spec, or (if not expressable in openapi) as a list of status messages on the address space resource:

* Checking that a valid `plan` and `type` is set. `type` can be an OpenAPIv3 enum. `plan` must be checked by address-space-controller
* Checking that connector references and endpoints are valid, sufficient with OpenAPIv3.
* Attaching infraUuid label and annotation. Consider deriving the infraUuid based on namespace + name rather than a random UUID. This will prevent the situation where user can remove the annotation and label on the CR.

=== Address

The `Address` resource is used by the address-space-controller (for quota checks), standard-controller and agent. These components will have to be updated to support using CRDs in the same way as for the `AddressSpace` resource.

The following validations are performed on an `Address` resource:

* Checking that a valid `plan` and `type` is set. `type` can be an OpenAPIv3 enum but must also be checked in per-address-space-type controllers. `plan` must be checked by standard-controller/agent
* Checking that forwarders point to a valid address space connector. Must be checked by standard-controller.
* Checking that name is prefixed with a valid address space name. Must be checked by standard-controller/agent.

=== MessagingUser

The messaging-user is only used by the API server today, which uses keycloak to persist users.

With CRDs, a new 'messaginguser-controller' component is needed (added to the enmasse-controller-manager for instance). This
component will sync messaginguser resources with Keycloak.

Having the `MessagingUser` as a CRDs means that we can no longer 'hide' the password for the `password` authentication type. An option of reading passwords from secrets should be provided when creating messaging users.

Validation on the `MessagingUser` can mostly be done by OpenAPIv3 validation, but name prefix checking for address space name must be done by controller.

== Upgrade process for address spaces and addresses

Upgrading EnMasse to a version that supports CRDs require that resources based on ConfigMaps must be converted to CRD-based resources, before the reconiliation loops start. Otherwise, it could end up with infrastructure being deleted.


The upgrade process will follow these steps:

1. Service admin will apply new bundle/run new ansible playbook/update operator

The rest of the upgrade process will be performed automatically once a new version of the enmasse-operator starts.

The enmasse-operator will follow these steps at startup:

1. Detect that upgrade is needed (check address-space-controller deployment annotation?)
1. Scale down the address-space-controller and admin deployments to 0 and wait for pods to be deleted
1. Remove api-service for enmasse.io api group
1. Deploy new version of address space controller and scale up to 1


The upgraded address-space-controller will follow these steps at startup:

1. Read all AddressSpace configmaps and convert them to CRs
1. Read all Address configmaps and convert them to CRs
1. Delete all configmaps for AddressSpace and Address
1. Upgrade standard-controller and agent 

Conversion of the completion can be checked using the following commands:

```
kubectl get configmap -l type=address-space | wc -l
kubectl get addressspaces --all-namespaces=true | wc -l
```

For addresses:

```
kubectl get configmap -l type=address-config | wc -l
kubectl get addresses --all-namespaces=true | wc -l
```

In addition, the `enmasse-opereator`, `address-space-controller`, `admin` and `agent` logs should be checked for errors.

== Upgrade process for messaging users

The upgrade process will follow these steps:

1. Service admin will apply new bundle/run new ansible playbook/update operator

The rest of the upgrade process will be performed automatically once a new version of the enmasse-operator starts.

The enmasse-operator will follow these steps at startup:

1. Check if upgrade is needed
1. Remove api-service for user.enmasse.io api group
1. List all messaging users in all Keycloak realms and create corresponding MessagingUser resources
1. Start messaginguser-controller component that sync messaging users back to Keycloak
1. Start normal operation (update authentication service as normal)


== Tasks

It is recommended that the move to CRDs are done in 2 phases, one for each api group. This means the first phase will convert AddressSpace and Address to CRDs, and the second phase will convert MessagingUser to a CRD.


== Testing

* Upgrade tests will be important to verify that conversion is working as expected
* All current tests should pass without change (other than those checking validation that cannot be performed by OpenAPIv3, those need to check resource status instead), as that would potentially mean an API breakage

== Documentation

* Documentation should contain the upgrade procedure needed
