conanfile.py that accompanies this document is nonstandard, and not
backward compatible with
conan clients unless
Three key workflows require “glue” code to connect them with the upstream binary repositories that store the device images. The main conern of this document is the design and implementtation of that glue code.
The workflows in question are:
f0cal farm image import–
f0cal farm instance create–
f0cal my-device image instaall–
The data flow for each of these follows a typical pattern:
If it isn’t already there, the image binary is pulled into local cache by the binary manager.
The cache location of the binary is exposed to the glue code.
The glue code calls the appropriate orchestration entrypoint.
In our case, the binary manager is
conan and the orchestration entrypoints are
The remainder of this document presents a proposal for a concise implementation
of the above. We define a
f0cal.glue.conan namespace to hold the parts of the
implementation that generalize adequetly. Workflow-specific portions of the
implementation that live in
f0cal/my-device are called out as such.
One of the key aspects of our proposal is heavy reuse of the
separating the data from the actions associated therewith. The mechanism for
achieving this is provided by
conan itself in the form of
YAML blob that automatically gets ingested alongside the
name: raspbian-lite history: - version: 10 url: https://downloads.raspberrypi.org/raspbian_lite_latest hash_type: sha256 device_types: - raspberry-pi __default__: True - name: 2019-09-26 date: 2019-09-26 hash: a50237c2f718bd8d806b96df5b9d2174ce8b789eda1f03434ed2213bbca6c6ff device_types: - raspberry-pi/3bp - raspberry-pi/3b - name: 2019-06-20 date: 2019-06-20 hash: 2db6753cd9f40d0228b7c6d1f98a1298d65505487e7ade7e383f438990081bd8 url: http://debian.rutgers.edu/raspbian_images/raspberrypi/images/raspbian/2019-06-20-raspbian-stretch/2019-06-20-raspbian-buster-lite.zip
Where direct manipulation of the
conanfile is challenging, the above YAML is
machine-readable and can be edited by adjecent tooling.
More on the semantics embedded in the
conandata in future sections.
conanfile.py that supports the above
conandata is minimal.
The only thing it must do is inherit from
from f0cal.glue.conan import ConanFile as _ConanFile_ class ConanFile(_ConanFile): pass
The “business logic” required to consume the above
conandata is supplied by an
ConanFile class in the
f0cal.glue.conan namespace. Its purpose is
to derive as many data members as possible from the
The user is encouraged to treat this class as a direct proxy for
conans.ConanFile. For it to work correctly, we must assume that the
namespace is available on any
conan client using a
We believe that this is a reasonable assumption, though it does imply that
f0cal.glue.conan recipes are not backwards compatible with vanilla
from conans import ConanFile as _ConanFile class ConanFile(_ConanFile): _DEFAULT_INDEX = -1 _SPECIAL_PLACE = "img" _FIELD_TO_INDEX = "history" # Note 1 _INDEX_VAR = "F0CAL_INDEX" # Note 2 _DEFAULT_TEST = lambda _d: "__default__" in _d and _d["__default__"] == True # Note 3 generators = ["f0cal"] requires = "f0cal-generator/0.1@f0cal/testing", # Note 2 @property @lru_cache() def index(self): # Note 3 return int(os.environ.get(self._INDEX_VAR, self._DEFAULT_INDEX)) def _conan_data(self, idx): _ = self.conan_data["defaults"] _.update(self.conan_data["history"][idx]) return types.SimpleNamespace(**_) @property def name(self): # Note 4 self.conan_data["name"] @property def version(self): # Note 5 self._conan_data(self.index).version def build(self): _d = self._conan_data(self.index) dargs = dict([(_d.hash_type, _d.hash)]) tools.get(_d.url, destination="img", **dargs) def package(self): self.move("img") def package_info(self): pass
1 The primary field of interest in the YAML is defined here.
2. We index into the
history field using an environment variable. The name of
that environment variable can be changed here. (An environment variable is
not an ideal solution, but it is
currently the only option
3. To save typing, defaults for the
history entries can be defined in a
special entry. This entry is culled from the final
_INDEX_VAR is applied.
4. The only field that is not pulled from the
history is the
version field is pulled from the
Generators are the mechanism provided by
conan for exposing the local binary
package cache to downstream tooling. Several default generators exist to support
tooling such as
f0cal provides a generator that renders
the package configuration to YAML.
import os from conans.models import Generator as _Generator class Generator(_Generator): _GEN_PATH_VAR = "F0CAL_OUT_PATH" # Note 1 _DEFAULT_GEN_PATH = os.path.join(os.cwd(), "f0cal.yml") # Note 2 @property def filename(self): return os.environ.get(self._GEN_PATH_VAR, self._DEFAULT_GEN_PATH)
Again, we use a environment variable to control the filesystem destination of the YAML outuput.
If the named environment variable is not supplied, the default output destination is the current working directory.
The interface between the YAML output of the generator, above, and the
orchestration code is a single
class that models what the binary can do next.
class Image: ... @classmethod def from_yaml(cls, yaml_path): ... def install(self, **kwargs): cmd = ["salt-call", "--local", "state.sls", "f0cal.my-device.image.install"] return self._saltbox_run(cmd, pillar=self.pillar_json, saltenv="f0cal-public") def test(self, **kwargs): ... def import(self, **kwargs): ... def overlay(self, **kwargs): cmd = ["salt-run", "state.orchestrate", "f0cal."] return self._saltbox_run(cmd, pillar=self.pillar_json, saltenv="f0cal-private") @classmethod def _saltbox_run(cls, cmd, pillar, saltenv=None, **kwargs) config = saltbox.SaltBoxConfig.from_env() with saltbox.SaltBox.executor_factory(config) as api: api.execute(cmd)
@f0cal.entrypoint(['my-device', 'image', 'install'], args=_image_install_args) def _image_install(parser, core, image, device_type): ref = ConanFileReference.loads(image, validate=False) info = Conan().install_reference(ref, generators=["f0cal"]) image_obj = Image.from_yaml(info.f0cal["output_path"]) return image_obj.install()