toxicbuild.master package

Submodules

toxicbuild.master.build module

class toxicbuild.master.build.Build(*args, **kwargs)[source]

Bases: mongoengine.document.EmbeddedDocument

A set of steps for a repository. This is the object that stores the build data. The build is carried by the slave.

PENDING = 'pending'
STATUSES = ['running', 'fail', 'success', 'exception', 'warning', 'pending']
branch

A unicode string field.

builder
finished

Datetime field.

Uses the python-dateutil library if available alternatively use time.strptime to parse the dates. Note: python-dateutil’s parser is fully featured and when installed you can utilise it to convert varying types of date formats into valid python datetime objects.

Note: Microseconds are rounded to the nearest millisecond.
Pre UTC microsecond support is effectively broken. Use ComplexDateTimeField if you need accurate microsecond support.
get_buildset()[source]

Returns the buildset that ‘owns’ this build.

named_tree

A unicode string field.

output

The build output. It is the commands + the output of the steps.

repository
slave
started

Datetime field.

Uses the python-dateutil library if available alternatively use time.strptime to parse the dates. Note: python-dateutil’s parser is fully featured and when installed you can utilise it to convert varying types of date formats into valid python datetime objects.

Note: Microseconds are rounded to the nearest millisecond.
Pre UTC microsecond support is effectively broken. Use ComplexDateTimeField if you need accurate microsecond support.
status

A unicode string field.

steps
to_dict(id_as_str=False)[source]
to_json()[source]
update()[source]

Does an atomic update in this embedded document.

uuid

A UUID field.

New in version 0.6.

class toxicbuild.master.build.BuildManager(repository)[source]

Bases: toxicbuild.core.utils.LoggerMixin

Controls which builds should be executed sequentially or in parallel.

add_builds(revisions)[source]

Adds the builds for a given revision in the build queue.

Parameters:revision – A list of toxicbuild.master.RepositoryRevision instances for the build.
add_builds_for_slave(buildset, slave, builders=[])[source]

Adds builds for a given slave on a given buildset.

Parameters:
  • buildset – An instance of toxicbuild.master.BuildSet.
  • slaves – An instance of toxicbuild.master.Slave.
  • builders – A list of toxicbuild.master.Builder. If not builders all builders for this slave and revision will be used.
build_queues
connect2signals()[source]

Connects the BuildManager to the revision_added signal.

get_builders(slave, revision)[source]

Get builders for a given slave and revision.

Parameters:
is_building
start_pending()[source]

Starts all pending buildsets that are not already scheduled for self.repository.

class toxicbuild.master.build.BuildSet(*args, **kwargs)[source]

Bases: toxicbuild.master.build.SerializeMixin, mongomotor.document.Document

A list of builds associated with a revision.

exception DoesNotExist

Bases: mongoengine.errors.DoesNotExist

exception BuildSet.MultipleObjectsReturned

Bases: mongoengine.errors.MultipleObjectsReturned

BuildSet.PENDING = 'pending'
BuildSet.author

A unicode string field.

BuildSet.branch

A unicode string field.

BuildSet.builds
BuildSet.commit

A unicode string field.

BuildSet.commit_date

Datetime field.

Uses the python-dateutil library if available alternatively use time.strptime to parse the dates. Note: python-dateutil’s parser is fully featured and when installed you can utilise it to convert varying types of date formats into valid python datetime objects.

Note: Microseconds are rounded to the nearest millisecond.
Pre UTC microsecond support is effectively broken. Use ComplexDateTimeField if you need accurate microsecond support.
classmethod BuildSet.create(repository, revision, save=True)[source]

Creates a new buildset.

Parameters:
  • repository – An instance of toxicbuild.master.Repository.
  • revision – An instance of toxicbuild.master.RepositoryRevision.
  • save – Indicates if the instance should be saved to database.
BuildSet.created

Datetime field.

Uses the python-dateutil library if available alternatively use time.strptime to parse the dates. Note: python-dateutil’s parser is fully featured and when installed you can utilise it to convert varying types of date formats into valid python datetime objects.

Note: Microseconds are rounded to the nearest millisecond.
Pre UTC microsecond support is effectively broken. Use ComplexDateTimeField if you need accurate microsecond support.
BuildSet.get_builds_for(builder=None, branch=None)[source]
BuildSet.get_pending_builds()[source]
BuildSet.get_status()[source]
BuildSet.id

A field wrapper around MongoDB’s ObjectIds.

BuildSet.objects = QuerySet
BuildSet.repository
BuildSet.revision
BuildSet.title

A unicode string field.

BuildSet.to_dict(id_as_str=False)[source]
BuildSet.to_json()[source]
class toxicbuild.master.build.BuildStep(*args, **kwargs)[source]

Bases: mongoengine.document.EmbeddedDocument

A step for a build. This is the object that will store the step data. Who actually execute the steps is the slave.

STATUSES = ['running', 'fail', 'success', 'exception', 'warning']
command

A unicode string field.

finished

Datetime field.

Uses the python-dateutil library if available alternatively use time.strptime to parse the dates. Note: python-dateutil’s parser is fully featured and when installed you can utilise it to convert varying types of date formats into valid python datetime objects.

Note: Microseconds are rounded to the nearest millisecond.
Pre UTC microsecond support is effectively broken. Use ComplexDateTimeField if you need accurate microsecond support.
index

32-bit integer field.

name

A unicode string field.

output

A unicode string field.

started

Datetime field.

Uses the python-dateutil library if available alternatively use time.strptime to parse the dates. Note: python-dateutil’s parser is fully featured and when installed you can utilise it to convert varying types of date formats into valid python datetime objects.

Note: Microseconds are rounded to the nearest millisecond.
Pre UTC microsecond support is effectively broken. Use ComplexDateTimeField if you need accurate microsecond support.
status

A unicode string field.

to_dict()[source]
to_json()[source]
uuid

A UUID field.

New in version 0.6.

class toxicbuild.master.build.Builder(*args, **kwargs)[source]

Bases: toxicbuild.master.build.SerializeMixin, mongomotor.document.Document

The entity responsible for executing the build steps.

exception DoesNotExist

Bases: mongoengine.errors.DoesNotExist

exception Builder.MultipleObjectsReturned

Bases: mongoengine.errors.MultipleObjectsReturned

classmethod Builder.create(**kwargs)[source]

Creates a new Builder.

Parameters:kwargs – kwargs passed to the builder constructor
classmethod Builder.get(**kwargs)[source]

Returns a builder instance.

classmethod Builder.get_or_create(**kwargs)[source]

Returns a builder instance. If it does not exist, creates it.

Parameters:kwargs – kwargs to match the builder.
Builder.get_status()[source]

Returns the builder status.

Builder.id

A field wrapper around MongoDB’s ObjectIds.

Builder.name

A unicode string field.

Builder.objects = QuerySet
Builder.repository
Builder.to_dict(id_as_str=False)[source]

Returns a dictionary for this builder.

Parameters:id_as_str – If true, the object id will be converted to string
Builder.to_json()[source]

Returns a json for this builder.

class toxicbuild.master.build.SerializeMixin[source]

Bases: object

Simple mixin to serialization relatad stuff.

async_to_json()[source]

Async version of to_json. Expects a to_dict coroutine.

to_dict(id_as_str=False)[source]

Transforms a Document into a dictionary.

Parameters:id_as_str – If true, transforms the id field into a string.

toxicbuild.master.client module

class toxicbuild.master.client.BuildClient(slave, *args, **kwargs)[source]

Bases: toxicbuild.core.client.BaseToxicClient

A client to toxicbuild.slave.server.BuildServer

build(build, process_coro=None)[source]

Requests a build for the build server.

Parameters:
  • build – The build that will be executed.
  • process_coro – A coroutine to process the intermediate build information sent by the build server.
healthcheck()[source]

Asks to know if the server is up and running

list_builders(repo_url, vcs_type, branch, named_tree)[source]

Asks the server for the builders available for repo_url, on branch and named_tree.

toxicbuild.master.client.get_build_client(slave, addr, port)[source]

Instanciate toxicbuild.master.client.BuildClient and connects it to a build server

toxicbuild.master.exceptions module

exception toxicbuild.master.exceptions.CloneException[source]

Bases: Exception

exception toxicbuild.master.exceptions.DBError[source]

Bases: Exception

exception toxicbuild.master.exceptions.UIFunctionNotFound[source]

Bases: Exception

toxicbuild.master.hole module

class toxicbuild.master.hole.HoleHandler(data, action, protocol)[source]

Bases: object

Handles the incomming connections for the UIHole. It has the following methods available to the clients:

  • repo-add
  • repo-get
  • repo-list
  • repo-remove
  • repo-update
  • repo-add-slave
  • repo-remove-slave
  • repo-add-branch
  • repo-remove-branch
  • repo-enable-plugin
  • repo-disable-plugin
  • repo-start-build
  • slave-add
  • slave-get
  • slave-list
  • slave-remove
  • slave-update
  • plugins-list
  • buildset-list
  • builder-show
  • list-funcs
builder_list(**kwargs)[source]

List builders.

Parameters:kwargs – Arguments to filter the list.
builder_show(repo_name, builder_name, skip=0, offset=None)[source]

Returns information about one specific builder.

Parameters:
  • repo_name – The builder’s repository name.
  • builder_name – The bulider’s name.
  • skip – How many elements we should skip in the result.
  • offset – How many results we should return.
buildset_list(repo_name=None, skip=0, offset=None)[source]

Lists all buildsets.

If repo_name, only builders from this repository will be listed. :param repo_name: Repository’s name. :param skip: skip for buildset list. :param offset: offset for buildset list.

handle()[source]
list_funcs()[source]

Lists the functions available for user interfaces.

plugins_list()[source]

Lists all plugins available to the master.

repo_add(repo_name, repo_url, update_seconds, vcs_type, slaves=None)[source]

Adds a new repository and first_run() it.

Parameters:
  • repo_name – Repository name
  • repo_url – Repository vcs url
  • update_seconds – Time to poll for changes
  • vcs_type – Type of vcs being used.
  • slaves – A list of slave names.
repo_add_branch(repo_name, branch_name, notify_only_latest=False)[source]

Adds a branch to the list of branches of the repository.

Parameters:
  • repo_name – Reporitory name
  • branch_name – Branch’s name
Notify_only_latest:
 

If True only the latest commit in the branch will trigger a build.

repo_add_slave(repo_name, slave_name)[source]

Adds a slave to a repository.

Parameters:
  • repo_name – Repository name.
  • slave_name – Slave name.
repo_disable_plugin(repo_name, **kwargs)[source]

Disables a plugin from a repository.

Parameters:
  • repo_name – Repository name.
  • kwargs – kwargs passed to the plugin
repo_enable_plugin(repo_name, plugin_name, **kwargs)[source]

Enables a plugin to a repository.

Parameters:
  • repo_name – Repository name.
  • plugin_name – Plugin name
  • kwargs – kwargs passed to the plugin.
repo_get(repo_name=None, repo_url=None)[source]

Shows information about one specific repository. One of repo_name or repo_url is required.

Parameters:
  • repo_name – Repository name,
  • repo_url – Repository vcs url.
repo_list()[source]

Lists all repositories.

repo_remove(repo_name)[source]

Removes a repository from toxicubild.

Parameters:repo_name – Repository name.
repo_remove_branch(repo_name, branch_name)[source]

Removes a branch from the list of branches of a repository. :param repo_name: Repository name :param branch_name: Branch’s name.

repo_remove_slave(repo_name, slave_name)[source]

Removes a slave from toxicbuild.

repo_start_build(repo_name, branch, builder_name=None, named_tree=None, slaves=[])[source]

Starts a(some) build(s) in a given repository.

repo_update(repo_name, **kwargs)[source]

Updates repository information.

Parameters:
  • repo_name – Repository name
  • kwargs – kwargs to update the repository
slave_add(slave_name, slave_host, slave_port, slave_token)[source]

Adds a new slave to toxicbuild.

slave_get(slave_name)[source]

Returns information about on specific slave

slave_list()[source]

Lists all slaves.

slave_remove(slave_name)[source]

Removes a slave from toxicbuild.

slave_update(slave_name, **kwargs)[source]

Updates infomation of a slave.

class toxicbuild.master.hole.HoleServer(addr='127.0.0.1', port=6666)[source]

Bases: object

get_protocol_instance()[source]
serve()[source]
class toxicbuild.master.hole.UIStreamHandler(protocol)[source]

Bases: toxicbuild.core.utils.LoggerMixin

Handler that keeps the connection open and messages when builds and steps are stated or finished.

build_added(reciever, **kw)[source]
build_finished(reciever, **kw)[source]
build_started(reciever, **kw)[source]
handle()[source]
send_info(info_type, build=None, step=None)[source]
send_repo_status_info(repo, old_status, new_status)[source]

Called by the signal repo_status_changed

Parameters:
  • repo – The repository that had its status changed.
  • old_status – The old status of the repository
  • new_status – The new status of the repostiory.
send_step_output_info(repo, step_info)[source]

Called by the signal step_output_arrived.

Parameters:
  • repo – The repository that is building something.
  • step_info – The information about the step output.
step_finished(reciever, **kw)[source]
step_started(reciever, **kw)[source]

toxicbuild.master.plugins module

This module implements plugins meat to be used in reaction to some signal sent by the master in the build process.

To implement your own plugins you must to subclass toxicbuild.master.plugins.MasterPlugin and implement a run() method.

The class toxicbuild.master.plugins.MasterPlugin is a mongomotor’s document that you can subclass and create your own fields to store the plugin’s config params. It already has the following fields:

Example:

import asyncio
from mongomotor.fields import StringField
from toxicbuild.master.plugins import MasterPlugin

class MyPlugin(MasterPlugin):

    name = 'my-plugin'
    type = 'notification'
    something_to_store_on_database = StringField()

    @asyncio.coroutine
    def run(self):
        '''Here is where you implement your stuff'''
class toxicbuild.master.plugins.MasterPlugin(*args, **kwargs)[source]

Bases: toxicbuild.core.plugins.Plugin, mongoengine.document.EmbeddedDocument

Base plugin for master’s plugins. Master’s plugins usually react to signals sent by the master.

branches
classmethod get_schema(to_serialize=False)[source]

Returns a dictionary with the schema of the plugin.

name = 'BaseMasterPlugin'
run()[source]

Runs the plugin. You must implement this in your plugin.

statuses
stop()[source]

Stops the plugin. Here is where you may disconnect from signals or other stuff needed to stop your plugin.

to_dict()[source]
type = None
class toxicbuild.master.plugins.MetaMasterPlugin[source]

Bases: mongoengine.base.metaclasses.DocumentMetaclass

Metaclass that sets name and type to the class definition as mongo fields while keeping the interface of setting your plugin’s name and type as string in definition time.

class toxicbuild.master.plugins.SlackPlugin(*args, **kwargs)[source]

Bases: toxicbuild.master.plugins.MasterPlugin

Plugin that send notifications about builds to slack.

channel_name

A unicode string field.

name = 'slack-notification'
run()[source]
send_finished_msg(repo, build)[source]

Sends a message about a finished build to a slack channel.

Parameters:build – A build that just finished.
send_started_msg(repo, build)[source]

Sends a message about a started build to a slack channel.

Parameters:build – A build that just started.
stop()[source]
type = 'notification'
webhook_url

A field that validates input as an URL.

New in version 0.3.

toxicbuild.master.pollers module

class toxicbuild.master.pollers.Poller(repository, vcs_type, workdir)[source]

Bases: toxicbuild.core.utils.LoggerMixin

Class to poll changes from a vcs, process them and notificate about incoming changes

is_polling()[source]
log(msg, level='info')[source]
notify_change(*revisions)[source]

Notify about new revisions added to the repository.

Parameters:revisions – A list of new revisions
poll()[source]

Check for changes on repository and if there are changes, notify about it.

process_changes()[source]

Process all changes since the last revision in db

toxicbuild.master.repository module

class toxicbuild.master.repository.Repository(*args, **kwargs)[source]

Bases: mongomotor.document.Document, toxicbuild.core.utils.LoggerMixin

Repository is where you store your code and where toxicbuild looks for incomming changes.

exception DoesNotExist

Bases: mongoengine.errors.DoesNotExist

exception Repository.MultipleObjectsReturned

Bases: mongoengine.errors.MultipleObjectsReturned

Repository.add_builds_for_slave(buildset, slave, builders=[])[source]

Adds a buildset to the build queue of a given slave for this repository.

Parameters:
Repository.add_or_update_branch(branch_name, notify_only_latest=False)[source]

Adds a new branch to this repository. If the branch already exists updates it with a new value.

Parameters:
  • branch_name – The name of a branch
  • notify_only_latest – If we should build only the most recent build of this branch
Repository.add_revision(branch, commit, commit_date, author, title)[source]

Adds a revision to the repository.

Parameters:
  • commit – commit uuid
  • branch – branch name
  • commit_date – commit’s date (on authors time)
Repository.add_slave(slave)[source]

Adds a new slave to a repository.

Parameters:slave – A slave instance.
Repository.branches
Repository.clone_status

A unicode string field.

classmethod Repository.create(name, url, update_seconds, vcs_type, slaves=None, branches=None)[source]

Creates a new repository and schedule it.

Parameters:
  • name – Repository name.
  • url – Repository version control system url
  • update_seconds – How long we should wait until poll the changes again.
  • vcs_type – Which type of version control system this repository uses.
  • slaves – A list of slaves for this repository.
  • branches – A list of branches config for this repository.
Repository.disable_plugin(**kwargs)[source]

Disables a plugin to the repository.

Parameters:kwargs – kwargs to match the plugin.
Repository.enable_plugin(plugin_name, **plugin_config)[source]

Enables a plugin to this repository.

Parameters:
  • plugin_name – The name of the plugin that is being enabled.
  • plugin_config – A dictionary containing the configuration passed to the plugin.
classmethod Repository.get(**kwargs)[source]

Returns a repository instance

Parameters:kwargs – kwargs to match the repository.
Repository.get_known_branches()[source]

Returns the names for the branches that already have some revision here.

Repository.get_latest_revision_for_branch(branch)[source]

Returns the latest revision for a given branch

Parameters:branch – branch name
Repository.get_latest_revisions()[source]

Returns the latest revision for all known branches

Repository.get_status()[source]

Returns the status for the repository. The status is the status of the last buildset created for this repository that is not pending.

Repository.id

A field wrapper around MongoDB’s ObjectIds.

Repository.name

A unicode string field.

Repository.objects = QuerySet
Repository.plugins
Repository.poller
Repository.remove()[source]

Removes all builds and builders and revisions related to the repository, removes the poller from the scheduler, removes the source code from the file system and then removes the repository.

Repository.remove_branch(branch_name)[source]

Removes a branch from this repository.

Parameters:branch_name – The branch name.
Repository.remove_slave(slave)[source]

Removes a slave from a repository.

Parameters:slave – A slave instance.
Repository.schedule()[source]

Schedules all needed actions for a repository. The actions are:

  • Update source code using self.update_code
  • Starts builds that are pending using self.build_manager.start_pending.
  • Connects to build_started and build_finished signals to handle changing of status.
classmethod Repository.schedule_all()[source]

Schedule all repositories.

Repository.slaves
Repository.to_dict(id_as_str=False)[source]
Repository.update_code()[source]

Updates the repository’s code. It is just a wrapper for self.poller.poll, so I can handle exceptions here.

Repository.update_seconds

32-bit integer field.

Repository.url

A unicode string field.

Repository.vcs_type

A unicode string field.

Repository.workdir

The directory where the source code of this repository is cloned into

class toxicbuild.master.repository.RepositoryBranch(*args, **kwargs)[source]

Bases: mongoengine.document.EmbeddedDocument

name

A unicode string field.

notify_only_latest

Boolean field type.

New in version 0.1.2.

to_dict()[source]
class toxicbuild.master.repository.RepositoryRevision(*args, **kwargs)[source]

Bases: mongomotor.document.Document

A commit in the code tree.

exception DoesNotExist

Bases: mongoengine.errors.DoesNotExist

exception RepositoryRevision.MultipleObjectsReturned

Bases: mongoengine.errors.MultipleObjectsReturned

RepositoryRevision.author

A unicode string field.

RepositoryRevision.branch

A unicode string field.

RepositoryRevision.commit

A unicode string field.

RepositoryRevision.commit_date

Datetime field.

Uses the python-dateutil library if available alternatively use time.strptime to parse the dates. Note: python-dateutil’s parser is fully featured and when installed you can utilise it to convert varying types of date formats into valid python datetime objects.

Note: Microseconds are rounded to the nearest millisecond.
Pre UTC microsecond support is effectively broken. Use ComplexDateTimeField if you need accurate microsecond support.
classmethod RepositoryRevision.get(**kwargs)[source]
RepositoryRevision.id

A field wrapper around MongoDB’s ObjectIds.

RepositoryRevision.objects = QuerySet
RepositoryRevision.repository
RepositoryRevision.title

A unicode string field.

toxicbuild.master.scheduler module

A very simple implementation of a in memory task scheduler using asyncio.

class toxicbuild.master.scheduler.PeriodicTask(call_or_coro, interval)[source]

Bases: object

A task that will be executed in a periodic time interval

class toxicbuild.master.scheduler.TaskScheduler[source]

Bases: object

A simple scheduler for periodic tasks.

add(call_or_coro, interval)[source]

Adds call_or_coro to be consumed at interval.

Parameters:
  • call_or_coro – callable or coroutine to be consumed.
  • interval – time in seconds to consume call_or_coro.
consume_tasks()[source]
remove(call_or_coro)[source]

Removes call_or_coro from the scheduler :param call_or_coro: callable or coroutine to remove

remove_by_hash(cc_hash)[source]

Removes the callable or couroutine scheduled using cc_hash.

start()[source]
stop()[source]

toxicbuild.master.signals module

toxicbuild.master.slave module

class toxicbuild.master.slave.Slave(*args, **kwargs)[source]

Bases: mongomotor.document.Document, toxicbuild.core.utils.LoggerMixin

Slaves are the entities that actualy do the work of execute steps. The comunication to slaves is through the network (using toxicbuild.master.client.BuildClient). The steps are actually decided by the slave.

exception DoesNotExist

Bases: mongoengine.errors.DoesNotExist

exception Slave.MultipleObjectsReturned

Bases: mongoengine.errors.MultipleObjectsReturned

Slave.build(build)[source]

Connects to a build server and requests a build on that server

Parameters:build – An instance of toxicbuild.master.build.Build
classmethod Slave.create(**kwargs)[source]
classmethod Slave.get(**kwargs)[source]
Slave.get_client()[source]

Returns a toxicbuild.master.client.BuildClient instance already connected to the server.

Slave.healthcheck()[source]

Check if the build server is up and running

Slave.host

A unicode string field.

Slave.id

A field wrapper around MongoDB’s ObjectIds.

Slave.is_alive

Boolean field type.

New in version 0.1.2.

Slave.list_builders(revision)[source]

List builder available in for a given revision

Parameters:revision – An instance of toxicbuild.master.repository.RepositoryRevision
Slave.name

A unicode string field.

Slave.objects = QuerySet
Slave.port

32-bit integer field.

Slave.to_dict(id_as_str=False)[source]
Slave.token

A unicode string field.

Module contents

toxicbuild.master.create(root_dir)[source]

Creates a new toxicmaster environment.

Parameters:--root_dir – Root directory for toxicmaster.
toxicbuild.master.create_scheduler()[source]
toxicbuild.master.create_settings_and_connect()[source]
toxicbuild.master.ensure_indexes()[source]
toxicbuild.master.restart(workdir, pidfile='toxicmaster.pid')[source]

Restarts toxicmaster

The instance of toxicmaster in workdir will be restarted. :param workdir: Workdir for master to be killed. :param –pidfile: Name of the file to use as pidfile. Defaults to toxicmaster.pid

toxicbuild.master.run(loglevel)[source]

Runs Toxicbuild master

toxicbuild.master.start(workdir, daemonize=False, stdout='toxicmaster.log', stderr='toxicmaster.log', conffile=None, loglevel='info', pidfile='toxicmaster.pid')[source]

Starts toxicmaster.

Parameters:
  • workdir – Work directory for server.
  • --daemonize – Run as daemon. Defaults to False
  • --stdout – stdout path. Defaults to /dev/null
  • --stderr – stderr path. Defaults to /dev/null
  • --conffile (-c,) – path to config file. Defaults to None. If not conffile, will look for a file called toxicmaster.conf inside workdir
  • --loglevel – Level for logging messages. Defaults to info.
  • --pidfile – Name of the file to use as pidfile. Defaults to toxicslave.pid
toxicbuild.master.stop(workdir, pidfile='toxicmaster.pid')[source]

Kills toxicmaster.

Parameters:
  • --workdir – Workdir for master to be killed. Looks for a file toxicmaster.pid inside workdir.
  • --pidfile – Name of the file to use as pidfile. Defaults to toxicslave.pid
toxicbuild.master.toxicinit()[source]

Initialize services.