Skip to content

Adding support for a new Stackbuild specific schema #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Adding support for a new Stackbuild specific schema #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,14 @@ install:
install -m 644 README.rst \
${buildroot}${docdir}/python-kiwi_stackbuild_plugin/README

kiwi_stackbuild_plugin/schema.rng: kiwi_stackbuild_plugin/schema.rnc
# whenever the schema is changed this target will convert
# the short form of the RelaxNG schema to the format used
# in code and auto generates the python data structures
@type -p trang &>/dev/null || \
(echo "ERROR: trang not found in path: $(PATH)"; exit 1)
trang -I rnc -O rng kiwi_stackbuild_plugin/schema.rnc kiwi_stackbuild_plugin/schema.rng

build: clean tox
# create setup.py variant for rpm build.
# delete module versions from setup.py for building an rpm
Expand Down
30 changes: 30 additions & 0 deletions kiwi_stackbuild_plugin/defaults.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@
# along with kiwi-stackbuild. If not, see <http://www.gnu.org/licenses/>
#
import re
import importlib
from importlib.resources import as_file

from kiwi.defaults import Defaults
from typing import (
Expand Down Expand Up @@ -95,3 +97,31 @@ def get_container_config(
'author': maintainer
}
}

@staticmethod
def project_file(filename):
"""
Provides the python module base directory search path

The method uses the importlib.resources.path method to identify
files and directories from the application

:param string filename: relative project file

:return: absolute file path name

:rtype: str
"""
with as_file(importlib.resources.files('kiwi_stackbuild_plugin')) as path:
return f'{path}/{filename}'

@staticmethod
def get_schema_file():
"""
Provides file path to kiwi RelaxNG schema

:return: file path

:rtype: str
"""
return StackBuildDefaults.project_file('schema.rng')
7 changes: 7 additions & 0 deletions kiwi_stackbuild_plugin/exceptions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,10 @@ class KiwiStackBuildPluginRootSyncFailed(KiwiError):
Exception raised if the rsync process to sync the stash into
the root-tree failed
"""


class KiwiStackBuildPluginSchemaValidationFailed(KiwiError):
"""
Exception raised if the provided XML description is not compliant with
the stack build image schema
"""
9 changes: 9 additions & 0 deletions kiwi_stackbuild_plugin/kiwi-fake-schema.rnc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#==========================================
# Fake rnc file used only to convert
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidcassany This is really great work 👍 👍

The only part I struggle with is that fake schema. as stackbuild has a kiwi requirement you should have kiwi in the devenv as well as in the system as a dependency later on. In kiwi I developed a method named project_file which allows to reference any file that belongs to the project. The schema file belongs to the project and can be looked up as follows

from kiwi.defaults import Defaults
print(Defaults.project_file('schema/kiwi.rnc'))

On my system with a pip installed kiwi this gives

python3.11 lala.py 
/home/ms/.local/lib/python3.11/site-packages/kiwi/schema/kiwi.rnc

ls -l /home/ms/.local/lib/python3.11/site-packages/kiwi/schema/kiwi.rnc
-rw-r--r-- 1 ms users 141916 Jan 10 14:28 /home/ms/.local/lib/python3.11/site-packages/kiwi/schema/kiwi.rnc

Shouldn't this work for your case too ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In kiwi I developed a method named project_file which allows to reference any file that belongs to the project.

I saw it and this was the key to actually solve the schema validation, I am using it to dynamically replace the fake included file with the actual KIWI schema that is used at runtime. This was the key to solve the schema validation issues I had in the past.

The only reason to exist of this kiwi-fake-schema is to actually run make kiwi_stackbuild_plugin/schema.rng which in turn calls trang -I rnc -O rng <inputfile> <outputfile>. The problem is that it runs a nested syntax conversion, so it also parses and converts any included file in the input.rnc file.

So calling

make kiwi_stackbuild_plugin/schema.rng

also reads kiwi_stackbuild_plugin/kiwi-fake-schema.rnc and creates a kiwi_stackbuild_plugin/kiwi-fake-schema.rng. Using a real kiwi path here in this context is tricky, as this is just trying to convert from RNC format to RNG nothing else, I don't feel like running a dynamic path discovery here which requires active python development environment is what we want here, this is more a build/packaging thing than a runtime concern. Moreover there is the risk of overwriting and messing with conversion of the included RNC file (the actual kiwi schema from the system). I bet there might be better ways of building and managing it. Probably another option would be to simply forget about RNC and use directly the RNG XML file as the schema, since it is a simple schema I think it should be doable too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's an option to write RNG directly. The RNC is the so called compact form of a RelaxNG schema. Most people use the compact form because it's easier to read and to maintain. But in this special case it might be ok to work with the RNG format and avoid the conversion

# stackbuild schema.rnc file to schema.rng
# without requiring the actual KIWI schema
# file
#
start =
## The start pattern of an image
k.image
16 changes: 16 additions & 0 deletions kiwi_stackbuild_plugin/kiwi-fake-schema.rng
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
<?xml version="1.0" encoding="UTF-8"?>
<grammar xmlns:a="http://relaxng.org/ns/compatibility/annotations/1.0" xmlns="http://relaxng.org/ns/structure/1.0">
<!--
==========================================
Fake rnc file used only to convert
stackbuild schema.rnc file to schema.rng
without requiring the actual KIWI schema
file

-->
<start>
<ref name="k.image">
<a:documentation>The start pattern of an image</a:documentation>
</ref>
</start>
</grammar>
56 changes: 56 additions & 0 deletions kiwi_stackbuild_plugin/schema.rnc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#================
# FILE : schema.rnc
#****************
# PROJECT : KIWI - Stack Build Plugin
# COPYRIGHT : (c) 2021 SUSE LINUX Products GmbH
# :
# AUTHOR : David Cassany <[email protected]>
# :
# BELONGS TO : Operating System images
# :
# DESCRIPTION : This is the RELAX NG Schema for KIWI Rebuild Images
# : plugin configuration files. The schema is maintained
# : in the relax compact syntax. Any changes should
# : made in !! *** schema.rnc *** !!
# :
# :
# STATUS : Development
#****************

namespace rng = "http://relaxng.org/ns/structure/1.0"

# The real include value is computed and replaced in memory at runtime
# to match the actual KIWI schema of the KIWI module being loaded
include "kiwi-fake-schema.rnc" {
k.image.schemaversion.attribute =
## The allowed Schema version (fixed value)
attribute schemaversion { "0.1" }

k.image.attlist = k.image.name.attribute
& k.image.stackbuild.attribute
& k.image.displayname.attribute?
& k.image.id?
& k.image.schemaversion.attribute
& ( k.image.noNamespaceSchemaLocation.attribute?
| k.image.schemaLocation.attribute? )?

k.image =
## The root element of the configuration file
element image {
k.image.attlist &
k.description? &
k.preferences* &
k.profiles? &
k.users* &
k.drivers* &
k.strip* &
k.repository* &
k.packages*
}
}

div{
k.image.stackbuild.attribute =
## Identifies description as a stackbuild XML
attribute stackbuild { "true" }
}
98 changes: 98 additions & 0 deletions kiwi_stackbuild_plugin/schema.rng
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
================
FILE : schema.rnc
****************
PROJECT : KIWI - Stack Build Plugin
COPYRIGHT : (c) 2021 SUSE LINUX Products GmbH
:
AUTHOR : David Cassany <[email protected]>
:
BELONGS TO : Operating System images
:
DESCRIPTION : This is the RELAX NG Schema for KIWI Rebuild Images
: plugin configuration files. The schema is maintained
: in the relax compact syntax. Any changes should
: made in !! *** schema.rnc *** !!
:
:
STATUS : Development
****************
-->
<grammar xmlns:a="http://relaxng.org/ns/compatibility/annotations/1.0" xmlns:rng="http://relaxng.org/ns/structure/1.0" xmlns="http://relaxng.org/ns/structure/1.0">
<!--
The real include value is computed and replaced in memory at runtime
to match the actual KIWI schema of the KIWI module being loaded
-->
<include href="kiwi-fake-schema.rng">
<define name="k.image.schemaversion.attribute">
<attribute name="schemaversion">
<a:documentation>The allowed Schema version (fixed value)</a:documentation>
<value>0.1</value>
</attribute>
</define>
<define name="k.image.attlist">
<interleave>
<ref name="k.image.name.attribute"/>
<ref name="k.image.stackbuild.attribute"/>
<optional>
<ref name="k.image.displayname.attribute"/>
</optional>
<optional>
<ref name="k.image.id"/>
</optional>
<ref name="k.image.schemaversion.attribute"/>
<optional>
<choice>
<optional>
<ref name="k.image.noNamespaceSchemaLocation.attribute"/>
</optional>
<optional>
<ref name="k.image.schemaLocation.attribute"/>
</optional>
</choice>
</optional>
</interleave>
</define>
<define name="k.image">
<element name="image">
<a:documentation>The root element of the configuration file</a:documentation>
<interleave>
<ref name="k.image.attlist"/>
<optional>
<ref name="k.description"/>
</optional>
<zeroOrMore>
<ref name="k.preferences"/>
</zeroOrMore>
<optional>
<ref name="k.profiles"/>
</optional>
<zeroOrMore>
<ref name="k.users"/>
</zeroOrMore>
<zeroOrMore>
<ref name="k.drivers"/>
</zeroOrMore>
<zeroOrMore>
<ref name="k.strip"/>
</zeroOrMore>
<zeroOrMore>
<ref name="k.repository"/>
</zeroOrMore>
<zeroOrMore>
<ref name="k.packages"/>
</zeroOrMore>
</interleave>
</element>
</define>
</include>
<div>
<define name="k.image.stackbuild.attribute">
<attribute name="stackbuild">
<a:documentation>Identifies description as a stackbuild XML</a:documentation>
<value>true</value>
</attribute>
</define>
</div>
</grammar>
79 changes: 52 additions & 27 deletions kiwi_stackbuild_plugin/tasks/system_stackbuild.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@
import logging
from mock import patch
from docopt import docopt
from tempfile import TemporaryDirectory
from typing import (
Dict, List
)
Expand All @@ -82,9 +83,10 @@
from kiwi.utils.sync import DataSync
from kiwi.defaults import Defaults

from kiwi_stackbuild_plugin.xml_merge import XMLMerge
from kiwi_stackbuild_plugin.exceptions import (
KiwiStackBuildPluginTargetDirExists,
KiwiStackBuildPluginRootSyncFailed
KiwiStackBuildPluginRootSyncFailed,
)

log = logging.getLogger('kiwi')
Expand Down Expand Up @@ -150,34 +152,57 @@ def process(self) -> None:
)

if self.command_args.get('--description'):
with patch.object(
sys, 'argv', self._validate_kiwi_build_command(
[
'system', 'build',
'--description', self.command_args['--description'],
'--target-dir', self.command_args['--target-dir'],
'--allow-existing-root'
]
)
):
kiwi_task = SystemBuildTask(
should_perform_task_setup=False
)
merger = XMLMerge(self.command_args['--description'])
if merger.is_stackbuild_description():
merger.validate_schema()
with TemporaryDirectory(
prefix='kiwi_description.'
) as temp_desc:
merger.merge_description(
f'{image_root_dir}/image', temp_desc
)
self._kiwi_build_task(
temp_desc, self.command_args['--target-dir']
).process()

else:
self._kiwi_build_task(
self.command_args['--description'],
self.command_args['--target-dir']
).process()
else:
with patch.object(
sys, 'argv', self._validate_kiwi_create_command(
[
'system', 'create',
'--root', image_root_dir,
'--target-dir', self.command_args['--target-dir']
]
)
):
kiwi_task = SystemCreateTask(
should_perform_task_setup=False
)
self._kiwi_create_task(
image_root_dir, self.command_args['--target-dir']
).process()

kiwi_task.process()
def _kiwi_build_task(self, description: str, target_dir: str) -> SystemBuildTask:
with patch.object(
sys, 'argv', self._validate_kiwi_build_command(
[
'system', 'build',
'--description', description,
'--target-dir', target_dir,
'--allow-existing-root'
]
)
):
return SystemBuildTask(
should_perform_task_setup=False
)

def _kiwi_create_task(self, root_dir: str, target_dir: str) -> SystemCreateTask:
with patch.object(
sys, 'argv', self._validate_kiwi_create_command(
[
'system', 'create',
'--root', root_dir,
'--target-dir', target_dir
]
)
):
return SystemCreateTask(
should_perform_task_setup=False
)

def _validate_kiwi_create_command(
self, kiwi_create_command: List[str]
Expand Down
4 changes: 3 additions & 1 deletion kiwi_stackbuild_plugin/tasks/system_stash.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,9 @@ def process(self) -> None:
)
description = XMLDescription(kiwi_description)
xml_state = XMLState(
xml_data=description.load()
description.load(),
self.global_args['--profile'],
self.global_args['--type']
)
contact_info = xml_state.get_description_section()
image_name = self.command_args['--container-name'] or \
Expand Down
Loading
Loading