toxicbuild.master package

Submodules

toxicbuild.master.build module

class toxicbuild.master.build.Build(*args, **kwargs)[source]

Bases: mongoengine.document.EmbeddedDocument

A set of steps for a repository. This is the object that stores the build data. The build is carried by the slave.

coroutine cancel()[source]

Cancel the build if it is not started yet.

coroutine classmethod get(uuid)[source]

Returns a build based on a uuid.

Parameters:uuid – The uuid of the build.
coroutine get_buildset()[source]

Returns the buildset that ‘owns’ this build.

coroutine notify(event_type)[source]

Send a notification to the build_notification exchange informing about event_type

Parameters:event_type – The name of the event.
to_dict(id_as_str=False)[source]

Transforms the object into a dictionary.

Parameters:id_as_str – Indicates if the id should be a string or an ObjectId instance.
to_json()[source]

Returns a json representation of the buld.

coroutine update()[source]

Does an atomic update in this embedded document.

CANCELLED = 'cancelled'
CONFIG_TYPES = ['py', 'yaml']
PENDING = 'pending'
STATUSES = ['running', 'fail', 'success', 'exception', 'warning', 'pending', 'cancelled']
branch

The branch of the code that will be tested.

builder

A reference to an instance of Builder.

external

A reference to ExternalRevisionIinfo

finished

When the build was finished. It must be in UTC.

named_tree

A identifier of the commit, a sha, a tag name, etc…

output

The build output. It is the commands + the output of the steps.

repository

A referece to the Repository that owns the build

slave

A reference to the Slave that will execute the build.

started

When the build was started. It must be in UTC.

status

The current status of the build. May be on of the values in STATUSES.

steps

A list of BuildStep

total_time

The total time of the build execution.

uuid

An uuid that identifies the build

class toxicbuild.master.build.BuildManager(repository)[source]

Bases: toxicbuild.core.utils.LoggerMixin

Controls which builds should be executed sequentially or in parallel.

coroutine add_builds(revisions)[source]

Adds the builds for a given revision in the build queue.

Parameters:revision – A list of toxicbuild.master.RepositoryRevision instances for the build.
coroutine add_builds_for_slave(buildset, slave, builders=None)[source]

Adds builds for a given slave on a given buildset.

Parameters:
  • buildset – An instance of toxicbuild.master.BuildSet.
  • slaves – An instance of toxicbuild.master.Slave.
  • builders – A list of toxicbuild.master.Builder. If not builders all builders for this slave and revision will be used.
coroutine cancel_build(build_uuid)[source]

Cancel a given build.

Parameters:build_uuid – The uuid that indentifies the build to be cancelled.
coroutine cancel_previous_pending(buildset)[source]

Cancels the builds previous to buildset.

Parameters:buildset – An instance of BuildSet.
coroutine get_builders(slave, revision)[source]

Get builders for a given slave and revision.

Parameters:
coroutine start_pending()[source]

Starts all pending buildsets that are not already scheduled for self.repository.

build_queues

Returns the build queues for a repository.

is_building

Indicates if has some active build for a repository.

class toxicbuild.master.build.BuildSet(*args, **kwargs)[source]

Bases: toxicbuild.master.build.SerializeMixin, mongomotor.document.Document

A list of builds associated with a revision.

exception DoesNotExist

Bases: mongoengine.errors.DoesNotExist

exception MultipleObjectsReturned

Bases: mongoengine.errors.MultipleObjectsReturned

coroutine classmethod create(repository, revision)[source]

Creates a new buildset.

Parameters:
  • repository – An instance of toxicbuild.master.Repository.
  • revision – An instance of toxicbuild.master.RepositoryRevision.
  • save – Indicates if the instance should be saved to database.
coroutine get_builds_for(builder=None, branch=None)[source]

Returns the builds for a specific builder and/or branch.

Parameters:
  • builder – An instance of Builder.
  • branch – The name of the branch.
get_pending_builds()[source]

Returns the pending builds of the buildset.

get_status()[source]

Returns the status of the BuildSet

coroutine notify(event_type, status=None)[source]

Notifies an event to the build_notifications exchange.

Parameters:
  • event_type – The event type to notify about
  • status – The status of the buildset. If None, the return of get_status() will be used.
to_dict(id_as_str=False)[source]

Returns a dict representation of the object

to_json()[source]

Returns a json representation of the object.

PENDING = 'pending'
author

Commit author’s name.

branch

The branch of the commit

builds

A list of Build intances.

commit

The identifier of the commit that generated the buildset.

commit_date

The date of the commit

created

When the BuildSet was first created. It must be in UTC.

finished

When the buildset finished. It must be in UTC.

id

A field wrapper around MongoDB’s ObjectIds.

objects = QuerySet
repository

A referece to the Repository that owns the buildset

revision

A reference to the RepositoryRevision that generated this buildset.

started

When the BuildSet started to run. It must be in UTC.

title

Commit title

total_time

The total time spent in the buildset

class toxicbuild.master.build.BuildStep(*args, **kwargs)[source]

Bases: mongoengine.document.EmbeddedDocument

A step for a build. This is the object that will store the step data. Who actually execute the steps is the slave.

to_dict()[source]

Returns a dict representation of the BuildStep.

to_json()[source]

Returns a json representation of the BuildStep.

STATUSES = ['running', 'fail', 'success', 'exception', 'warning']
command

The command that executes the step

finished

When the step finished. It msut be in UTC.

index

The index of the step in the build.

name

The name of the step. Will be displayed in the ui.

output

The output of the step

started

When the step stated. It msut be in UTC.

status

The current status of the step. May be one of the values in STATUSES`

total_time

The total time spen in the step.

uuid

The uuid that indentifies the build step

class toxicbuild.master.build.Builder(*args, **kwargs)[source]

Bases: toxicbuild.master.build.SerializeMixin, mongomotor.document.Document

The entity responsible for executing the build steps.

exception DoesNotExist

Bases: mongoengine.errors.DoesNotExist

exception MultipleObjectsReturned

Bases: mongoengine.errors.MultipleObjectsReturned

coroutine classmethod create(**kwargs)[source]

Creates a new Builder.

Parameters:kwargs – kwargs passed to the builder constructor
coroutine classmethod get(**kwargs)[source]

Returns a builder instance.

coroutine classmethod get_or_create(**kwargs)[source]

Returns a builder instance. If it does not exist, creates it.

Parameters:kwargs – kwargs to match the builder.
coroutine get_status()[source]

Returns the builder status.

coroutine to_dict(id_as_str=False)[source]

Returns a dictionary for this builder.

Parameters:id_as_str – If true, the object id will be converted to string
coroutine to_json()[source]

Returns a json for this builder.

id

A field wrapper around MongoDB’s ObjectIds.

name

The name of the builder.

objects = QuerySet
repository

A referece to the Repository that owns the builder

class toxicbuild.master.build.SerializeMixin[source]

Bases: object

Simple mixin to serialization relatad stuff.

coroutine async_to_json()[source]

Async version of to_json. Expects a to_dict coroutine.

to_dict(id_as_str=False)[source]

Transforms a Document into a dictionary.

Parameters:id_as_str – If true, transforms the id field into a string.

toxicbuild.master.client module

class toxicbuild.master.client.BuildClient(slave, *args, **kwargs)[source]

Bases: toxicbuild.core.client.BaseToxicClient

A client to toxicbuild.slave.server.BuildServer

coroutine build(build, process_coro=None)[source]

Requests a build for the build server.

Parameters:
  • build – The build that will be executed.
  • process_coro – A coroutine to process the intermediate build information sent by the build server.
coroutine healthcheck()[source]

Asks to know if the server is up and running

coroutine list_builders(repo_url, vcs_type, branch, named_tree)[source]

Asks the server for the builders available for repo_url, on branch and named_tree.

toxicbuild.master.client.get_build_client(slave, addr, port, use_ssl=True, validate_cert=True)[source]

Instanciate toxicbuild.master.client.BuildClient and connects it to a build server

toxicbuild.master.exceptions module

exception toxicbuild.master.exceptions.CloneException[source]

Bases: Exception

exception toxicbuild.master.exceptions.DBError[source]

Bases: Exception

exception toxicbuild.master.exceptions.ImpossibleCancellation[source]

Bases: Exception

exception toxicbuild.master.exceptions.InvalidCredentials[source]

Bases: Exception

exception toxicbuild.master.exceptions.NotEnoughPerms[source]

Bases: Exception

exception toxicbuild.master.exceptions.OwnerDoesNotExist[source]

Bases: Exception

exception toxicbuild.master.exceptions.RepoBranchDoesNotExist[source]

Bases: Exception

exception toxicbuild.master.exceptions.UIFunctionNotFound[source]

Bases: Exception

exception toxicbuild.master.exceptions.UnknownSchedulerAction[source]

Bases: Exception

toxicbuild.master.hole module

class toxicbuild.master.hole.HoleHandler(data, action, protocol)[source]

Bases: object

Handles the incomming connections for the UIHole. It has the following methods available to the clients:

  • repo-add
  • repo-get
  • repo-list
  • repo-remove
  • repo-update
  • repo-add-slave
  • repo-remove-slave
  • repo-add-branch
  • repo-remove-branch
  • repo-enable-plugin
  • repo-disable-plugin
  • repo-start-build
  • repo-cancel-build
  • slave-add
  • slave-get
  • slave-list
  • slave-remove
  • slave-update
  • plugins-list
  • plugin-get
  • buildset-list
  • builder-show
  • list-funcs
  • user-add
  • user-remove
  • user-authenticate
coroutine builder_list(**kwargs)[source]

List builders.

Parameters:kwargs – Arguments to filter the list.
coroutine builder_show(repo_name, builder_name, skip=0, offset=None)[source]

Returns information about one specific builder.

Parameters:
  • repo_name – The builder’s repository name.
  • builder_name – The bulider’s name.
  • skip – How many elements we should skip in the result.
  • offset – How many results we should return.
coroutine buildset_list(repo_name=None, skip=0, offset=None)[source]

Lists all buildsets.

If repo_name, only builders from this repository will be listed. :param repo_name: Repository’s name. :param skip: skip for buildset list. :param offset: offset for buildset list.

coroutine handle()[source]
list_funcs()[source]

Lists the functions available for user interfaces.

plugin_get(**kwargs)[source]

Returns a specific plugin.

plugins_list()[source]

Lists all plugins available to the master.

coroutine repo_add(repo_name, repo_url, owner_id, update_seconds, vcs_type, slaves=None, parallel_builds=None)[source]

Adds a new repository and first_run() it.

Parameters:
  • repo_name – Repository name
  • repo_url – Repository vcs url
  • owner_id – Id of the repository’s owner.
  • update_seconds – Time to poll for changes
  • vcs_type – Type of vcs being used.
  • slaves – A list of slave names.
Params parallel_builds:
 

How many parallel builds this repository executes. If None, there is no limit.

coroutine repo_add_branch(repo_name, branch_name, notify_only_latest=False)[source]

Adds a branch to the list of branches of the repository.

Parameters:
  • repo_name – Reporitory name
  • branch_name – Branch’s name
Notify_only_latest:
 

If True only the latest commit in the branch will trigger a build.

coroutine repo_add_slave(repo_name, slave_name)[source]

Adds a slave to a repository.

Parameters:
  • repo_name – Repository name.
  • slave_name – Slave name.
coroutine repo_cancel_build(repo_name_or_id, build_uuid)[source]

Cancels a build if possible.

Parameters:
  • repo_name_or_id – The name or the id of the repository.
  • buid_uuid – The uuid of the build to be cancelled.
coroutine repo_disable_plugin(repo_name, **kwargs)[source]

Disables a plugin from a repository.

Parameters:
  • repo_name – Repository name.
  • kwargs – kwargs passed to the plugin
coroutine repo_enable_plugin(repo_name, plugin_name, **kwargs)[source]

Enables a plugin to a repository.

Parameters:
  • repo_name – Repository name.
  • plugin_name – Plugin name
  • kwargs – kwargs passed to the plugin.
coroutine repo_get(repo_name=None, repo_url=None)[source]

Shows information about one specific repository. One of repo_name or repo_url is required.

Parameters:
  • repo_name – Repository name,
  • repo_url – Repository vcs url.
coroutine repo_list()[source]

Lists all repositories.

coroutine repo_remove(repo_name)[source]

Removes a repository from toxicubild.

Parameters:repo_name – Repository name.
coroutine repo_remove_branch(repo_name, branch_name)[source]

Removes a branch from the list of branches of a repository. :param repo_name: Repository name :param branch_name: Branch’s name.

coroutine repo_remove_slave(repo_name, slave_name)[source]

Removes a slave from toxicbuild.

coroutine repo_start_build(repo_name, branch, builder_name=None, named_tree=None, slaves=None)[source]

Starts a(some) build(s) in a given repository.

coroutine repo_update(repo_name, **kwargs)[source]

Updates repository information.

Parameters:
  • repo_name – Repository name
  • kwargs – kwargs to update the repository
coroutine slave_add(slave_name, slave_host, slave_port, slave_token, owner_id, use_ssl=True, validate_cert=True)[source]

Adds a new slave to toxicbuild.

Parameters:
  • slave_name – A name for the slave,
  • slave_host – Host where the slave is.
  • slave_port – Port to connect to the slave
  • slave_token – Auth token for the slave.
  • owner_id – Slave’s owner id.
  • use_ssl – Indicates if the slave uses a ssl connection.
Pram validate_cert:
 

Should the slave certificate be validated?

coroutine slave_get(slave_name)[source]

Returns information about one specific slave

coroutine slave_list()[source]

Lists all slaves.

coroutine slave_remove(slave_name)[source]

Removes a slave from toxicbuild.

coroutine slave_update(slave_name, **kwargs)[source]

Updates infomation of a slave.

coroutine user_add(email, password, allowed_actions, username=None)[source]

Adds a new user.

Parameters:
  • email – User email.
  • password – User password
  • allowed_actions – What the user can do.
  • username – Username for the user.
coroutine user_authenticate(username_or_email, password)[source]

Authenticates an user. Returns user.to_dict() is authenticated. Raises InvalidCredentials if a user with this credentials does not exist.

Parameters:
  • username_or_email – Username or email to use to authenticate.
  • password – Not encrypted password.
coroutine user_remove(**kwargs)[source]

Removes a user from the system.

class toxicbuild.master.hole.HoleServer(addr='127.0.0.1', port=6666, loop=None, use_ssl=False, **ssl_kw)[source]

Bases: toxicbuild.core.utils.LoggerMixin

A server that uses the UIHole protocol.

get_protocol_instance()[source]
serve()[source]
coroutine shutdown()[source]
sync_shutdown(signum=None, frame=None)[source]
class toxicbuild.master.hole.UIStreamHandler(protocol)[source]

Bases: toxicbuild.core.utils.LoggerMixin

Handler that keeps the connection open and messages when builds and steps are stated or finished.

coroutine build_added(sender, **kw)[source]
coroutine build_cancelled_fn(sender, **kw)[source]
coroutine build_finished(sender, **kw)[source]
coroutine build_started(sender, **kw)[source]
coroutine check_repo_added(msg)[source]
coroutine handle()[source]
coroutine send_info(info_type, sender, build=None, step=None)[source]
coroutine send_repo_added_info(message)[source]

Sends a message about a repository’s creation.

Parameters:message – A message from the repo_added exchange.
coroutine send_repo_status_info(message)[source]

Sends a message about a repository’s new status

Parameters:message – A message from the repo_status_changed exchange.
coroutine send_response(code, body)[source]
send_step_output_info(repo, step_info)[source]

Called by the signal step_output_arrived.

Parameters:
  • repo – The repository that is building something.
  • step_info – The information about the step output.
coroutine step_finished(sender, **kw)[source]
coroutine step_started(sender, **kw)[source]

toxicbuild.master.mail module

exception toxicbuild.master.mail.MailSenderNotConnected[source]

Bases: Exception

class toxicbuild.master.mail.MailSender(recipients)[source]

Bases: toxicbuild.core.utils.LoggerMixin

Simple mail sender. Takes host/port/auth params from settings.

To send an email, use the context manager:

recipients = ['me@mail.com', 'other@mail.com']
async with MailSender(recipients) as sender:
    await sender.send('Subject', 'This is the message body')
coroutine connect()[source]

Connects to a smtp server.

coroutine disconnect()[source]

Closes the connection to the smpt server

coroutine send(subject, message)[source]

Send an email message.

Parameters:
  • subject – Email subject.
  • message – Email body

toxicbuild.master.plugins module

This module implements plugins meant to be used in reaction to some notification sent by the master, usually in the build process.

To implement your own plugins you must to subclass toxicbuild.master.plugins.MasterPlugin and implement a run method.

The class MasterPlugin is a mongomotor’s document that you can subclass and create your own fields to store the plugin’s config params. It already has the following fields:

  • branches: A list of branch names that triggers the plugin.
  • statuses: A list of statuses that triggers the plugin.

Example:

from mongomotor.fields import StringField
from toxicbuild.master.plugins import MasterPlugin

class MyPlugin(MasterPlugin):

    # you must define name and type
    name = 'my-plugin'
    type = 'notification'
    # optionally you may define pretty_name and description
    pretty_name = "My Plugin"
    description = "A very cool plugin"

    something_to_store_on_database = PrettyStringField()

    async def run(self, sender, info):
        '''Here is where you implement your stuff.

        :param sender: A repository instance.
        :param info: A dictionary with some information for the
          plugin to handle.'''
class toxicbuild.master.plugins.CustomWebhookPlugin(*args, **kwargs)[source]

Bases: toxicbuild.master.plugins.NotificationPlugin

Sends a POST request to a custom URL. The request mime type is json/application and the body of the request has a json with 3 keys: repository, build and buildset.

coroutine send_finished_message(repo, build)[source]

Sends a message about a finished build. You must implement this in your plugin.

Parameters:
coroutine send_started_message(repo, build)[source]

Sends a message about a started build. You must implement this in your plugin.

Parameters:
description = 'Sends messages to a custom webhook.'
name = 'custom-webhook'
no_list = False
pretty_name = 'Custom Webhook'
type = 'notification'
webhook_url
class toxicbuild.master.plugins.EmailPlugin(*args, **kwargs)[source]

Bases: toxicbuild.master.plugins.NotificationPlugin

Sends notification about builds through email

coroutine send_finished_message(repo, build)[source]

Sends a message about a finished build. You must implement this in your plugin.

Parameters:
coroutine send_started_message(repo, build)[source]

Sends a message about a started build. You must implement this in your plugin.

Parameters:
description = 'Sends email messages'
name = 'email-notification'
no_list = False
pretty_name = 'Email'
recipients
type = 'notification'
class toxicbuild.master.plugins.MasterPlugin(*args, **kwargs)[source]

Bases: toxicbuild.core.utils.LoggerMixin, toxicbuild.core.plugins.Plugin, mongoengine.document.EmbeddedDocument

Base plugin for master’s plugins. Master’s plugins usually react to signals sent by the master.

classmethod get_schema(to_serialize=False)[source]

Returns a dictionary with the schema of the plugin.

classmethod list_for_event_type(event_type, no_events=False)[source]

Lists the plugins that react for a given event.

:param event_type. The event type to match against. :param no_events: Indicates if the plugins with no events should be listed. That may be used to make plugins without events react to all events.

coroutine run(sender, info)[source]

Runs the plugin. You must implement this in your plugin.

coroutine stop()[source]

Stops the plugin. Here is where you may disconnect from signals or other stuff needed to stop your plugin.

to_dict()[source]
description = "Base for master's plugins"
events = []
name = 'BaseMasterPlugin'
no_list = False
pretty_name = ''
type = None
uuid

A UUID field.

New in version 0.6.

class toxicbuild.master.plugins.MetaMasterPlugin[source]

Bases: toxicbuild.core.plugins.PluginMeta, mongoengine.base.metaclasses.DocumentMetaclass

Metaclass that sets name and type to the class definition as mongo fields while keeping the interface of setting your plugin’s name and type as string in definition time.

class toxicbuild.master.plugins.NotificationPlugin(*args, **kwargs)[source]

Bases: toxicbuild.master.plugins.MasterPlugin

coroutine run(sender, info)[source]

Executed when a notification about a build arrives. Reacts to builds that started or finished.

Parameters:
  • sender – An instance of Repository.
  • info – A dictionary with information about a build.
coroutine send_finished_message(repo, build)[source]

Sends a message about a finished build. You must implement this in your plugin.

Parameters:
coroutine send_started_message(repo, build)[source]

Sends a message about a started build. You must implement this in your plugin.

Parameters:
coroutine stop()[source]

Stops the plugin. Here is where you may disconnect from signals or other stuff needed to stop your plugin.

branches
description = 'Base plugin for notifications'
events = ['build-started', 'build-finished']
name = 'NotificationPlugin'
no_list = True
pretty_name = ''
sender = None
statuses
type = 'notification'
class toxicbuild.master.plugins.PrettyFieldMixin(*args, **kwargs)[source]

Bases: object

A field with a descriptive name for humans

class toxicbuild.master.plugins.PrettyListField(*args, **kwargs)[source]

Bases: toxicbuild.master.plugins.PrettyFieldMixin, mongomotor.fields.ListField

class toxicbuild.master.plugins.PrettyStringField(*args, **kwargs)[source]

Bases: toxicbuild.master.plugins.PrettyFieldMixin, mongoengine.fields.StringField

class toxicbuild.master.plugins.PrettyURLField(*args, **kwargs)[source]

Bases: toxicbuild.master.plugins.PrettyFieldMixin, mongoengine.fields.URLField

class toxicbuild.master.plugins.SlackPlugin(*args, **kwargs)[source]

Bases: toxicbuild.master.plugins.NotificationPlugin

Plugin that send notifications about builds to slack.

coroutine send_finished_message(repo, build)[source]

Sends a message about a finished build. You must implement this in your plugin.

Parameters:
coroutine send_started_message(repo, build)[source]

Sends a message about a started build. You must implement this in your plugin.

Parameters:
channel_name
description = 'Sends messages to a slack channel'
name = 'slack-notification'
no_list = False
pretty_name = 'Slack'
type = 'notification'
webhook_url

toxicbuild.master.pollers module

class toxicbuild.master.pollers.Poller(repository, vcs_type, workdir)[source]

Bases: toxicbuild.core.utils.LoggerMixin

Class to poll changes from a vcs, process them and notificate about incoming changes

coroutine external_poll(external_url, external_name, external_branch, into)[source]

Fetches the changes of a external (not the origin) repository into a local branch.

Parameters:
  • external_url – The url of the external remote repository.
  • external_name – The name to identiry the external repo.
  • external_branch – The name of the branch in the external repo.
  • into – The name of the local repository.
is_polling()[source]
log(msg, level='info')[source]

Appends the class name before the log message.

coroutine notify_change(*revisions)[source]

Notify about new revisions added to the repository.

Parameters:revisions – A list of new revisions
coroutine poll(repo_branches=None)[source]

Check for changes on repository and if there are changes, notify about it.

Parameters:repo_branches – Param to be passed to process_changes().
coroutine process_changes(repo_branches=None)[source]

Process all changes since the last revision in db

Parameters:repo_branches

The branches to look for incomming changes. If no branches, all branches in repo config will be used. It is a dictionary with the following format:

class toxicbuild.master.pollers.PollerServer(loop=None)[source]

Bases: toxicbuild.master.utils.BaseQueueReactorServer

A server for pollers. Uses Rabbitmq to publish/consume messages from the master

coroutine handle_update_request(msg)[source]

Handle an update code request sent by the master.

toxicbuild.master.repository module

class toxicbuild.master.repository.Repository(*args, **kwargs)[source]

Bases: toxicbuild.master.document.OwnedDocument, toxicbuild.core.utils.LoggerMixin

Repository is where you store your code and where toxicbuild looks for incomming changes.

exception DoesNotExist

Bases: mongoengine.errors.DoesNotExist

exception MultipleObjectsReturned

Bases: mongoengine.errors.MultipleObjectsReturned

coroutine add_builds_for_slave(buildset, slave, builders=None)[source]

Adds a buildset to the build queue of a given slave for this repository.

Parameters:
coroutine add_or_update_branch(branch_name, notify_only_latest=False)[source]

Adds a new branch to this repository. If the branch already exists updates it with a new value.

Parameters:
  • branch_name – The name of a branch
  • notify_only_latest – If we should build only the most recent build of this branch
coroutine add_revision(branch, commit, commit_date, author, title, body=None, external=None)[source]

Adds a revision to the repository.

Parameters:
  • commit – commit uuid
  • branch – branch name
  • commit_date – commit’s date (on authors time)
  • author – The author of the commit
  • title – The commit title.
  • body – The commit body.
  • external – Information about an external remote if the revision came from an external.
classmethod add_running_build()[source]

Add a running build to the count of running builds among all repositories.

coroutine add_slave(slave)[source]

Adds a new slave to a repository.

Parameters:slave – A slave instance.
coroutine bootstrap()[source]

Initialise the needed stuff. Schedules updates for code, start of pending builds, connect to signals.

coroutine classmethod bootstrap_all()[source]
coroutine cancel_build(build_uuid)[source]

Cancels a build.

Parameters:build_uuid – The uuid of the build.
coroutine classmethod create(**kwargs)[source]

Creates a new repository and schedule it.

Parameters:kwargs – kwargs used to create the repository.
coroutine disable_plugin(**kwargs)[source]

Disables a plugin to the repository.

Parameters:kwargs – kwargs to match the plugin.
coroutine enable_plugin(plugin_name, **plugin_config)[source]

Enables a plugin to this repository.

Parameters:
  • plugin_name – The name of the plugin that is being enabled.
  • plugin_config – A dictionary containing the plugin’s configuration.
coroutine classmethod get(**kwargs)[source]

Returns a repository instance and create locks if needed

Parameters:kwargs – kwargs to match the repository.
get_branch(branch_name)[source]

Returns an instance of RepositoryBranch

coroutine classmethod get_for_user(user, **kwargs)[source]

Returns a repository if user has permission for it. If not raises an error.

Parameters:
  • user – User who is requesting the repository.
  • kwargs – kwargs to match the repository.
coroutine get_known_branches()[source]

Returns the names for the branches that already have some revision here.

coroutine get_latest_revision_for_branch(branch)[source]

Returns the latest revision for a given branch

Parameters:branch – branch name
coroutine get_latest_revisions()[source]

Returns the latest revision for all known branches

get_plugins_for_event(event)[source]

Returns the plugins that react for a given event.

Parameters:event – The event for the plugins to react
classmethod get_running_builds()[source]

Returns the number of running builds among all the repos.

coroutine get_status()[source]

Returns the status for the repository. The status is the status of the last buildset created for this repository that is not pending.

get_url()[source]
log(msg, level='info')[source]

Appends the class name before the log message.

notify_only_latest(branch_name)[source]

Indicates if a branch notifies only the latest revision.

Parameters:branch_name – The name of the branch.
coroutine remove()[source]

Removes all builds and builders and revisions related to the repository, removes the poller from the scheduler, removes the source code from the file system and then removes the repository.

coroutine remove_branch(branch_name)[source]

Removes a branch from this repository.

Parameters:branch_name – The branch name.
classmethod remove_running_build()[source]

Removes a running build from the count of running builds among all repositories.

coroutine remove_slave(slave)[source]

Removes a slave from a repository.

Parameters:slave – A slave instance.
coroutine request_build(branch, builder_name=None, named_tree=None, slaves=None)[source]

Publishes a message in the repo_notifications exchange requesting a build. Uses the routing_key build-requested

coroutine request_removal()[source]

Request the removal of a repository by publishing a message in the repo_notifications queue with the routing key repo-removal-requested.

schedule()[source]

Schedules all needed actions for a repository. The actions are:

  • Sends an add-udpate-code to the scheduler server.
  • Starts builds that are pending using self.build_manager.start_pending.
  • Connects to build_started and build_finished signals to handle changing of status.
  • Runs the enabled plugins.
coroutine classmethod schedule_all()[source]

Schedule all repositories.

coroutine start_build(branch, builder_name=None, named_tree=None, slaves=None)[source]

Starts a (some) build(s) in the repository.

classmethod stop_consuming_messages()[source]

Informs that Repository should stop consumming messages from exchanges.

coroutine to_dict(id_as_str=False)[source]

Returns a dict representation of the object.

coroutine update_code(repo_branches=None, external=None, wait_for_lock=False)[source]

Requests a code update to a poller and waits for its response. This is done using update_code and poll_status exchanges.

Parameters:
  • repo_branches

    A dictionary with information about the branches to be updated. If no repo_branches all branches in the repo config will be updated.

    The dictionary has the following format.

    {'branch-name': notify_only_latest}
    
  • external – If we should update code from an external (not the origin) repository, external is the information about this remote repo.
  • wait_for_lock – Indicates if we should wait for the release of the lock or simply return if we cannot get a lock.
coroutine classmethod wait_build_requests()[source]

Waits for build requests that arrive in the repo_notifications exchange with the routing key build-requested

coroutine classmethod wait_removal_request()[source]

Waits for removal requests in the repo_notifications exchange with the routing key removal-requested

coroutine classmethod wait_revisions()[source]

Waits for messages sent by pollers about new revisions.

branches

A list of RepositoryBranch. These branches are the ones that trigger builds. If no branches, all branches will trigger builds.

clone_status

The status of the clone.

fetch_url

A url used to actually fetch the code. If using some kind of authentication based in a url, this may change often.

id

A field wrapper around MongoDB’s ObjectIds.

name

The name of the repository.

objects = QuerySet
parallel_builds

Max number of builds in parallel that this repo exeutes If None, there’s no limit for parallel builds.

plugins

The list of plugins enabled for this reposiory.

schedule_poller

Indicates if we should schedule periodical polls for changes in code. If the repo was imported from an external service that sends webhooks (or something else) this should be False.

slaves

A list of Slave. The slaves here are the slaves allowed to run builds for this repo.

update_seconds

If the repository added manually (not imported), indicates the inteval for polling for new revisions.

url

The url of the repository.

vcs

An instance of a subclass of VCS.

vcs_type

The type of vcs used in this repo.

workdir

The directory where the source code of this repository is cloned into

class toxicbuild.master.repository.RepositoryBranch(*args, **kwargs)[source]

Bases: mongoengine.document.EmbeddedDocument

The configuration for a branch of a repository.

to_dict()[source]

Returns a dict representation of the obj.

name

The name of the branch.

notify_only_latest

If True, only the latest revision will be notified and only the last revision will generate a buildset.

class toxicbuild.master.repository.RepositoryRevision(*args, **kwargs)[source]

Bases: mongomotor.document.Document

A commit in the code tree.

exception DoesNotExist

Bases: mongoengine.errors.DoesNotExist

exception MultipleObjectsReturned

Bases: mongoengine.errors.MultipleObjectsReturned

create_builds()[source]

Checks for instructions in the commit body to know if a revision should create builds.

Known instructions:

  • ci: skip - If in the commit body there’s this instruction,
    no builds will be created for this revision. The regex for match this instruction is #\s*ci:\s*skip(\s+|$)
coroutine classmethod get(**kwargs)[source]

Returs a RepositoryRevision object.

coroutine to_dict()[source]

Returns a dict representation of the object.

author

The author of the commit.

body

The commit body.

branch

The name of the revison branch.

commit

The identifier of the revision, a sha, a tag name, etc…

commit_date

Commit’s date.

external

A list of RepositoryRevisionExternal.

id

A field wrapper around MongoDB’s ObjectIds.

objects = QuerySet
repository

A referece to Repository.

title

The title of the commit.

toxicbuild.master.scheduler module

A very simple implementation of an in memory task scheduler using asyncio.

class toxicbuild.master.scheduler.PeriodicTask(call_or_coro, interval)[source]

Bases: object

A task that will be executed in a periodic time interval

class toxicbuild.master.scheduler.SchedulerServer(loop=None)[source]

Bases: toxicbuild.master.utils.BaseQueueReactorServer

Simple server to add or remove something from the scheduler.

coroutine handle_add_update_code(msg)[source]
coroutine handle_request(msg)[source]
coroutine handle_rm_update_code(msg)[source]
coroutine run()[source]

Starts the server

class toxicbuild.master.scheduler.TaskScheduler[source]

Bases: toxicbuild.core.utils.LoggerMixin

A simple scheduler for periodic tasks.

add(call_or_coro, interval)[source]

Adds call_or_coro to be consumed at interval.

Parameters:
  • call_or_coro – callable or coroutine to be consumed.
  • interval – time in seconds to consume call_or_coro.
consume_tasks()[source]
remove(call_or_coro)[source]

Removes call_or_coro from the scheduler :param call_or_coro: callable or coroutine to remove

remove_by_hash(cc_hash)[source]

Removes the callable or couroutine scheduled using cc_hash.

coroutine start()[source]
stop()[source]

toxicbuild.master.signals module

toxicbuild.master.slave module

class toxicbuild.master.slave.Slave(*args, **kwargs)[source]

Bases: toxicbuild.master.document.OwnedDocument, toxicbuild.core.utils.LoggerMixin

Slaves are the entities that actualy do the work of execute steps. The comunication to slaves is through the network (using toxicbuild.master.client.BuildClient). The steps are actually decided by the slave.

exception DoesNotExist

Bases: mongoengine.errors.DoesNotExist

exception MultipleObjectsReturned

Bases: mongoengine.errors.MultipleObjectsReturned

coroutine build(build)[source]

Connects to a build server and requests a build on that server

Parameters:build – An instance of toxicbuild.master.build.Build
coroutine classmethod create(**kwargs)[source]

Creates a new slave

coroutine classmethod get(**kwargs)[source]

Returns a slave instance.

coroutine get_client()[source]

Returns a BuildClient instance already connected to the server.

coroutine healthcheck()[source]

Check if the build server is up and running

coroutine list_builders(revision)[source]

List builder available in for a given revision

Parameters:revision – An instance of toxicbuild.master.repository.RepositoryRevision
to_dict(id_as_str=False)[source]

Returns a dict representation of the object.

host

Slave’s host.

id

A field wrapper around MongoDB’s ObjectIds.

is_alive

Indicates if the slave is up and running.

name

The name of the slave

objects = QuerySet
port

Port for the slave to listen.

token

Token for authentication.

use_ssl

Indicates if the build server in uses ssl connection.

validate_cert

Indicates if the certificate from the build server should be validated.

Module contents

toxicbuild.master.create(root_dir)[source]

Creates a new toxicmaster environment.

Parameters:--root_dir – Root directory for toxicmaster.
toxicbuild.master.create_scheduler()[source]
toxicbuild.master.create_settings_and_connect()[source]
toxicbuild.master.create_user(configfile, email=None, password=None, superuser=False)[source]

Creates a superuser in the master.

Parameters:
  • --email – User’s email.
  • --password – Password for authentication.
  • --superuser – Indicates if the user is a super user. Defaults to False
toxicbuild.master.ensure_indexes(*classes)[source]
toxicbuild.master.poller_server_init(server)[source]

Starts a poller server.

toxicbuild.master.restart(workdir, pidfile='toxicmaster.pid', loglevel='info')[source]

Restarts toxicmaster

The instance of toxicmaster in workdir will be restarted. :param workdir: Workdir for master to be killed. :param –pidfile: Name of the file to use as pidfile. Defaults to toxicmaster.pid :param –loglevel: Level for logging messages. Defaults to info.

toxicbuild.master.restart_poller(workdir, pidfile='toxicpoller.pid', loglevel='info')[source]

Restarts toxicmaster poller. The instance of toxicmaster poller in workdir will be restarted.

Parameters:
  • workdir – Workdir for the poller to be killed.
  • --pidfile – Name of the file to use as pidfile. Defaults to toxicpoller.pid
  • --loglevel – Level for logging messages. Defaults to info.
toxicbuild.master.restart_scheduler(workdir, pidfile='toxicscheduler.pid', loglevel='info')[source]

Restarts toxicmaster scheduler. The instance of toxicmaster scheduler in workdir will be restarted.

Parameters:
  • workdir – Workdir for the scheduler to be killed.
  • --pidfile – Name of the file to use as pidfile. Defaults to toxicscheduler.pid
  • --loglevel – Level for logging messages. Defaults to info.
toxicbuild.master.run(loglevel)[source]

Runs Toxicbuild master

toxicbuild.master.run_poller(loglevel)[source]

Runs a poller server for toxicbuild master.

toxicbuild.master.run_scheduler(loglevel)[source]

Runs a scheduler server for toxicbuild master.

toxicbuild.master.scheduler_server_init(server)[source]

Starts the scheduler server

toxicbuild.master.start(workdir, daemonize=False, stdout='toxicmaster.log', stderr='toxicmaster.log', conffile=None, loglevel='info', pidfile='toxicmaster.pid', no_scheduler=False, no_poller=False)[source]

Starts toxicmaster.

Parameters:
  • workdir – Work directory for server.
  • --daemonize – Run as daemon. Defaults to False
  • --stdout – stdout path. Defaults to /dev/null
  • --stderr – stderr path. Defaults to /dev/null
  • --conffile (-c,) – path to config file. Defaults to None. If not conffile, will look for a file called toxicmaster.conf inside workdir
  • --loglevel – Level for logging messages. Defaults to info.
  • --pidfile – Name of the file to use as pidfile. Defaults to toxicmaster.pid
toxicbuild.master.start_poller(workdir, daemonize=False, stdout='toxicpoller.log', stderr='toxicpoller.log', conffile=None, loglevel='info', pidfile='toxicpoller.pid')[source]

Starts a poller server.

Parameters:
  • workdir – Work directory for the server.
  • --daemonize – Run as daemon. Defaults to False
  • --stdout – stdout path. Defaults to /dev/null
  • --stderr – stderr path. Defaults to /dev/null
  • --conffile (-c,) – path to config file. Defaults to None. If not conffile, will look for a file called toxicmaster.conf inside workdir
  • --loglevel – Level for logging messages. Defaults to info.
  • --pidfile – Name of the file to use as pidfile. Defaults to toxicslave.pid
toxicbuild.master.start_scheduler(workdir, daemonize=False, stdout='toxicscheduler.log', stderr='toxicscheduler.log', conffile=None, loglevel='info', pidfile='toxicscheduler.pid')[source]

Starts a scheduler server.

Parameters:
  • workdir – Work directory for the server.
  • --daemonize – Run as daemon. Defaults to False
  • --stdout – stdout path. Defaults to /dev/null
  • --stderr – stderr path. Defaults to /dev/null
  • --conffile (-c,) – path to config file. Defaults to None. If not conffile, will look for a file called toxicmaster.conf inside workdir
  • --loglevel – Level for logging messages. Defaults to info.
  • --pidfile – Name of the file to use as pidfile. Defaults to toxicslave.pid
toxicbuild.master.stop(workdir, pidfile='toxicmaster.pid', kill=False)[source]

Kills toxicmaster.

Parameters:
  • --workdir – Workdir for master to be killed. Looks for a file toxicmaster.pid inside workdir.
  • --pidfile – Name of the file to use as pidfile. Defaults to toxicmaster.pid
  • kill – If true, send signum 9, otherwise, 15.
toxicbuild.master.stop_poller(workdir, pidfile='toxicpoller.pid', kill=False)[source]

Kills toxicmaster poller.

Parameters:
  • --workdir – Workdir for master to be killed. Looks for a file toxicmaster.pid inside workdir.
  • --pidfile – Name of the file to use as pidfile. Defaults to toxicpoller.pid
  • kill – If true, send signum 9, otherwise, 15.
toxicbuild.master.stop_scheduler(workdir, pidfile='toxicscheduler.pid', kill=False)[source]

Kills toxicmaster scheduler.

Parameters:
  • --workdir – Workdir for master to be killed. Looks for a file toxicmaster.pid inside workdir.
  • --pidfile – Name of the file to use as pidfile. Defaults to toxicscheduler.pid.
  • kill – If true, send signum 9, otherwise, 15.
toxicbuild.master.toxicinit(server)[source]

Initialize services.