91c4acf6

Note

The conanfile.py that accompanies this document is nonstandard, and not backward compatible with conan clients unless f0cal.glue.conan is installed.

Overview

Three key workflows require “glue” code to connect them with the upstream binary repositories that store the device images. The main conern of this document is the design and implementtation of that glue code.

The workflows in question are:

• f0cal farm image import –

• f0cal farm instance create –

• f0cal my-device image instaall –

The data flow for each of these follows a typical pattern:

1. If it isn’t already there, the image binary is pulled into local cache by the binary manager.

2. The cache location of the binary is exposed to the glue code.

3. The glue code calls the appropriate orchestration entrypoint.

In our case, the binary manager is conan and the orchestration entrypoints are managed by saltbox.

The remainder of this document presents a proposal for a concise implementation of the above. We define a f0cal.glue.conan namespace to hold the parts of the implementation that generalize adequetly. Workflow-specific portions of the implementation that live in f0cal/farm and f0cal/my-device are called out as such.

User interface

conandata.yml

One of the key aspects of our proposal is heavy reuse of the conanfile by separating the data from the actions associated therewith. The mechanism for achieving this is provided by conan itself in the form of conandata.yml, a YAML blob that automatically gets ingested alongside the conanfile.py at runtime.

name: raspbian-lite

history:

  - version: 10
    url: https://downloads.raspberrypi.org/raspbian_lite_latest
    hash_type: sha256
    device_types:
      - raspberry-pi
    __default__: True

  - name: 2019-09-26
    date: 2019-09-26
    hash: a50237c2f718bd8d806b96df5b9d2174ce8b789eda1f03434ed2213bbca6c6ff
    device_types:
      - raspberry-pi/3bp
      - raspberry-pi/3b

  - name: 2019-06-20
    date: 2019-06-20
    hash: 2db6753cd9f40d0228b7c6d1f98a1298d65505487e7ade7e383f438990081bd8
    url: http://debian.rutgers.edu/raspbian_images/raspberrypi/images/raspbian/2019-06-20-raspbian-stretch/2019-06-20-raspbian-buster-lite.zip

Where direct manipulation of the conanfile is challenging, the above YAML is machine-readable and can be edited by adjecent tooling.

More on the semantics embedded in the conandata in future sections.

conanfile.py

The user-defined conanfile.py that supports the above conandata is minimal. The only thing it must do is inherit from f0cal.glue.conan.ConaFile.

from f0cal.glue.conan import ConanFile as _ConanFile_

class ConanFile(_ConanFile):
    pass

f0cal.glue.conan.ConanFile

The “business logic” required to consume the above conandata is supplied by an overloaded ConanFile class in the f0cal.glue.conan namespace. Its purpose is to derive as many data members as possible from the conadata.yml.

The user is encouraged to treat this class as a direct proxy for conans.ConanFile. For it to work correctly, we must assume that the f0cal namespace is available on any conan client using a f0cal.glue.conan recipe. We believe that this is a reasonable assumption, though it does imply that f0cal.glue.conan recipes are not backwards compatible with vanilla conan recipes.

from conans import ConanFile as _ConanFile

class ConanFile(_ConanFile):
    _DEFAULT_INDEX = -1
    _SPECIAL_PLACE = "img"
    _FIELD_TO_INDEX = "history" # Note 1
    _INDEX_VAR = "F0CAL_INDEX" # Note 2
    _DEFAULT_TEST = lambda _d: "__default__" in _d and _d["__default__"] == True # Note 3
    
    generators = ["f0cal"]
    requires = "f0cal-generator/0.1@f0cal/testing", # Note 2

    @property
    @lru_cache()
    def index(self): # Note 3
        return int(os.environ.get(self._INDEX_VAR, self._DEFAULT_INDEX))

    def _conan_data(self, idx):
        _ = self.conan_data[]["defaults"]
        _.update(self.conan_data["history"][idx])
        return types.SimpleNamespace(**_)

    @property
    def name(self): # Note 4
        self.conan_data["name"]

    @property
    def version(self): # Note 5
        self._conan_data(self.index).version

    def build(self):
        _d = self._conan_data(self.index)
        dargs = dict([(_d.hash_type, _d.hash)])
        tools.get(_d.url, destination="img", **dargs)

    def package(self):
        self.move("img")

    def package_info(self):
        pass

Notes: 1 The primary field of interest in the YAML is defined here. 2. We index into the history field using an environment variable. The name of that environment variable can be changed here. (An environment variable is not an ideal solution, but it is currently the only option available.) 3. To save typing, defaults for the history entries can be defined in a special entry. This entry is culled from the final conandata dictionary before the _INDEX_VAR is applied. 4. The only field that is not pulled from the history is the name field. 5. The version field is pulled from the history.

f0cal.glue.conan.Generator

Generators are the mechanism provided by conan for exposing the local binary package cache to downstream tooling. Several default generators exist to support tooling such as cmake and scons. f0cal provides a generator that renders the package configuration to YAML.

import os
from conans.models import Generator as _Generator

class Generator(_Generator):

    _GEN_PATH_VAR = "F0CAL_OUT_PATH" # Note 1
    _DEFAULT_GEN_PATH = os.path.join(os.cwd(), "f0cal.yml") # Note 2

    @property
    def filename(self):
        return os.environ.get(self._GEN_PATH_VAR, self._DEFAULT_GEN_PATH)

Notes:

1. Again, we use a environment variable to control the filesystem destination of the YAML outuput.

2. If the named environment variable is not supplied, the default output destination is the current working directory.

f0cal.glue.conan.Image

The interface between the YAML output of the generator, above, and the orchestration code is a single class that models what the binary can do next.

class Image:
    ...
    @classmethod
    def from_yaml(cls, yaml_path):
        ...
        
    def install(self, **kwargs):
        cmd = ["salt-call", "--local", "state.sls", "f0cal.my-device.image.install"]
        return self._saltbox_run(cmd, pillar=self.pillar_json, saltenv="f0cal-public")
        
    def test(self, **kwargs):
        ...
        
    def import(self, **kwargs):
        ...
        
    def overlay(self, **kwargs):
        cmd = ["salt-run", "state.orchestrate", "f0cal."]
        return self._saltbox_run(cmd, pillar=self.pillar_json, saltenv="f0cal-private")
        
    @classmethod
    def _saltbox_run(cls, cmd, pillar, saltenv=None, **kwargs)
        config = saltbox.SaltBoxConfig.from_env()
        with saltbox.SaltBox.executor_factory(config) as api:
            api.execute(cmd)

f0cal.my-device

@f0cal.entrypoint(['my-device', 'image', 'install'], args=_image_install_args)
def _image_install(parser, core, image, device_type):
    ref = ConanFileReference.loads(image, validate=False)
    info = Conan().install_reference(ref, generators=["f0cal"])
    image_obj = Image.from_yaml(info.f0cal["output_path"])
    return image_obj.install()