toxicbuild.core package

Submodules

toxicbuild.core.build_config module

class toxicbuild.core.build_config.APTPluginConfig(conf)[source]

Bases: toxicbuild.core.build_config.BasePluginConfig

get_config()[source]
class toxicbuild.core.build_config.BasePluginConfig(conf)[source]

Bases: object

Plugin configs are meant to be used along with the LanguageConfig. Based in the toxicbuild.yml config we can create the build plugins configuration for a builder.

Note

The build plugins currently live in the slave but that is a very silly idea. In a future relase that’s gonna change.

class toxicbuild.core.build_config.LanguageConfig(conf)[source]

Bases: object

An abstraction to create builders based in a language config in a toxicbuild.yml config file.

DEFAULT_OS = 'debian'
LANGUAGE_PLUGINS = {'python': <class 'toxicbuild.core.build_config.PythonPluginConfig'>}
SYSTEM_PACKAGES_PLUGINS = {'debian': <class 'toxicbuild.core.build_config.APTPluginConfig'>, 'ubuntu': <class 'toxicbuild.core.build_config.APTPluginConfig'>}
builders
class toxicbuild.core.build_config.LanguagePluginConfig(lang_ver, conf)[source]

Bases: toxicbuild.core.build_config.BasePluginConfig

class toxicbuild.core.build_config.PythonPluginConfig(lang_ver, conf)[source]

Bases: toxicbuild.core.build_config.LanguagePluginConfig

get_config()[source]
toxicbuild.core.build_config.get_config(workdir, config_type, config_filename)[source]

Returns the build configuration for a given repository.

Parameters:
  • workdir – The directory where config file is located.
  • config_type – Which type of configuration we are using, ‘py’ or ‘yaml’.
  • config_filename – The filename of the config file.
toxicbuild.core.build_config.get_toxicbuildconf(directory)[source]

Returns the toxicbuild.conf module.

Parameters:directory – Directory to look for toxicbuild.conf
toxicbuild.core.build_config.get_toxicbuildconf_yaml(directory, filename='toxicbuild.yml')[source]

Returns the python objet representing the yaml build configuration.

Parameters:
  • directory – The path of the directory containing the config file.
  • filename – The actual name of the config file.
toxicbuild.core.build_config.list_builders_from_config(config, branch=None, slave=None, config_type='yml')[source]

Lists builders from a build config

Parameters:
  • config – The build configuration.
  • branch – The branch for which builders are being listed.
  • slave – The slave for which builders are being listed.

toxicbuild.core.client module

This module implements a base client for basic tcp communication, reading and writing json data.

Usage:

host = 'somehost.net'
port = 1234
async with BaseToxicClient(host, port):
    await client.write({'hello': 'world'})
    json_response = await client.get_response()
class toxicbuild.core.client.BaseToxicClient(host, port, loop=None, use_ssl=False, validate_cert=True, **ssl_kw)[source]

Bases: toxicbuild.core.utils.LoggerMixin

Base client for communication with toxicbuild servers.

coroutine connect()[source]

Connects to the server.

Note

This is called by the asynchronous context manager (aka async with)

disconnect()[source]

Disconnects from the server

coroutine get_response()[source]

Reads data from the server and raises and exception in case of error

is_connected()[source]
coroutine read()[source]

Reads data from the server. Expects a json.

coroutine write(data)[source]

Writes data to the server.

Parameters:data – Data to be sent to the server. Will be converted to json and enconded using utf-8.

toxicbuild.core.cmd module

class toxicbuild.core.cmd.ToxicProgram(*args, **kwargs)[source]

Bases: mando.core.Program

Transforms a python function into a command line program. Extends mando’s Program to execute _generate_command() only when execute() is called to enable toxicbuild command hacks.

execute(*args, **kwargs)[source]

Parse the arguments and execute the resulting command.

Parameters:args – The arguments to parse.

toxicbuild.core.conf module

class toxicbuild.core.conf.Settings(envvar, default_filename)[source]

Bases: object

Simple interface to a settings file.

toxicbuild.core.exceptions module

exception toxicbuild.core.exceptions.BadJsonData[source]

Bases: Exception

exception toxicbuild.core.exceptions.ConfigError[source]

Bases: Exception

exception toxicbuild.core.exceptions.ExecCmdError[source]

Bases: Exception

exception toxicbuild.core.exceptions.ImpossibillityError[source]

Bases: Exception

exception toxicbuild.core.exceptions.PluginNotFound[source]

Bases: Exception

exception toxicbuild.core.exceptions.ToxicClientException[source]

Bases: Exception

exception toxicbuild.core.exceptions.VCSError[source]

Bases: Exception

toxicbuild.core.exchange module

class toxicbuild.core.exchange.AmqpConnection(**conn_kwargs)[source]

Bases: toxicbuild.core.utils.LoggerMixin

A connection for a broker. We can have many channels over one connection.

coroutine connect()[source]

Connects to the Rabbitmq server.

coroutine disconnect()[source]

Disconnects from the Rabbitmq server.

class toxicbuild.core.exchange.Consumer(channel, queue, consumer_tag, nowait=False, timeout=0)[source]

Bases: asyncamqp.consumer.Consumer, toxicbuild.core.utils.LoggerMixin

coroutine fetch_message(cancel_on_timeout=True)[source]
class toxicbuild.core.exchange.Exchange(name, connection, exchange_type, durable=False, bind_publisher=True, exclusive_consumer_queue=False)[source]

Bases: toxicbuild.core.utils.LoggerMixin

A simple abstraction for an amqp exchange.

coroutine bind(routing_key, queue_name=None, channel=None)[source]

Binds the queue to the exchange.

Parameters:
  • routing_key – Routing key to bind the queue.
  • queue_name – The name of the queue to be bound. If not self.queue_name will be used.
  • channel – Optional channel to use in the communication. A new one will be used.
coroutine consume(wait_message=True, timeout=0, routing_key='', no_ack=False)[source]

Consumes a message from a Rabbitmq queue.

Parameters:
  • wait_message – Should we wait for new messages in the queue?
  • timeout – Timeout for waiting messages.
  • no_ack – Indicates if we should send a ack response to the server. The ack must be sent by the consumer.
coroutine declare(queue_name=None)[source]

Declares the exchange and queue.

Parameters:queue_name – The name for the queue to be declared. If None, self.queue_name will be used.
coroutine get_queue_size(queue_name=None)[source]
is_declared(queue_name=None)[source]
coroutine publish(message, routing_key='')[source]

Publishes a message to a Rabbitmq exchange

Parameters:
  • message – The message that will be published in the exchange. Must be something that can be serialized into a json.
  • routing_key – The routing key to pdublish the message.
coroutine queue_delete(queue_name=None)[source]

Deletes a queue. If not queue_name, self.queue_name will be used.

Parameters:queue_name – The name of the queue to be deleted.
coroutine unbind(routing_key, channel=None)[source]
class toxicbuild.core.exchange.JsonAckMessage(*args, **kwargs)[source]

Bases: asyncamqp.consumer.Message

coroutine acknowledge()[source]
coroutine reject(requeue=True)[source]

toxicbuild.core.exchanges module

toxicbuild.core.exchanges.create_exchanges(conn)[source]
toxicbuild.core.exchanges.disconnect_exchanges()[source]

toxicbuild.core.mail module

exception toxicbuild.core.mail.MailSenderNotConnected[source]

Bases: Exception

class toxicbuild.core.mail.MailSender(recipients, **smtp_settings)[source]

Bases: toxicbuild.core.utils.LoggerMixin

Simple mail sender.

To send an email, use the context manager:

recipients = ['me@mail.com', 'other@mail.com']
smtp_settings = {'host': 'some.host', 'port': 123,
                 'mail_from': 'me@myplace.net',
                 'username': 'me', 'password': '1234',
                 'validate_certs': True, 'starttls': False}

async with MailSender(recipients, **smtp_settigs) as sender:
    await sender.send('Subject', 'This is the message body')
coroutine connect()[source]

Connects to a smtp server.

coroutine disconnect()[source]

Closes the connection to the smpt server

coroutine send(subject, message)[source]

Send an email message.

Parameters:
  • subject – Email subject.
  • message – Email body

toxicbuild.core.plugins module

class toxicbuild.core.plugins.Plugin(*args, **kwargs)[source]

Bases: object

This is a base plugin. Plugins may implement aditional behavior to your builds.

classmethod get_plugin(name)[source]

Returns a Plugin subclass based on its name.

Parameters:name – Plugin’s name.
classmethod list_plugins(plugin_type=None)[source]

Returns a list of Plugin subclasses.

Parameters:plugin_type – the plugin’s type.
name = 'BaseCorePlugin'
no_list = False
class toxicbuild.core.plugins.PluginMeta[source]

Bases: type

toxicbuild.core.protocol module

class toxicbuild.core.protocol.BaseToxicProtocol(loop, connection_lost_cb=None)[source]

Bases: asyncio.streams.StreamReaderProtocol, toxicbuild.core.utils.LoggerMixin

Base protocol for toxicbulid servers

check_data()[source]

Checks if the data is valid, it means, checks if has some data, checks if it is a valid json and checks if it has a action key

client_connected()[source]

Coroutine that handles connections. You must implement this in your sub-classes. When this method is called, self.data, containing a dictionary with the data passed in the request and self.action, a string indicating which action to take are already available.

close_connection()[source]

Closes the connection with the client

connection_lost(exc)[source]

Called once, when the connection is lost.

Parameters:exc – The exception, if some.
connection_made(transport)[source]

Called once, when the client connects.

Parameters:transport – transport for asyncio.StreamReader and asyncio.StreamWriter.
get_json_data()[source]

Returns the json sent by the client.

get_raw_data()[source]

Returns the raw data sent by the client

send_response(code, body)[source]

Send a response to client formated by the (unknown) toxicbuild remote build specs.

Parameters:
  • code – code for this message. code == 0 is success and code > 0 is error.
  • body – response body. It has to be a serializable object.
encrypted_token = None

toxicbuild.core.requests module

This module implements a simple asynchronous interface for http requests.

Usage:

from toxicbuild.core import requests
response = yield from requests.get('http://google.com/')
print(response.text)
class toxicbuild.core.requests.Response(status, text, headers=None)[source]

Bases: object

Encapsulates a response from a http request

json()[source]

Loads the json in the response text.

toxicbuild.core.requests.delete(url, **kwargs)[source]

Performs a http DELETE request

Parameters:
  • url – Request’s url.
  • kwargs – Args passed to toxicbuild.core.requests._request().
toxicbuild.core.requests.get(url, **kwargs)[source]

Performs a http GET request

Parameters:
  • url – Request’s url.
  • kwargs – Args passed to toxicbuild.core.requests._request().
toxicbuild.core.requests.post(url, **kwargs)[source]

Performs a http POST request

Parameters:
  • url – Request’s url.
  • kwargs – Args passed to toxicbuild.core.requests._request().
toxicbuild.core.requests.put(url, **kwargs)[source]

Performs a http PUT request

Parameters:
  • url – Request’s url.
  • kwargs – Args passed to toxicbuild.core.requests._request().

toxicbuild.core.utils module

class toxicbuild.core.utils.LoggerMixin[source]

Bases: object

A simple mixin to use log on a class.

log(msg, level='info')[source]

Appends the class name before the log message.

classmethod log_cls(msg, level='info')[source]
class toxicbuild.core.utils.MatchKeysDict[source]

Bases: dict

A dictionary that returns the values matching the keys using toxicbuild.core.utils.match_string().

>>> d = MatchKeysDict()
>>> d['k*'] = 1
>>> d['key']
1
>>> k['keyboard']
1
get(k[, d]) → D[k] if k in D, else d. d defaults to None.[source]
class toxicbuild.core.utils.SettingsPatcher[source]

Bases: mongomotor.monkey.MonkeyPatcher

Patches the settings from pyrocumulus to use the same settings as toxibuild.

patch_pyro_settings(settings)[source]
class toxicbuild.core.utils.SourceSuffixesPatcher[source]

Bases: mongomotor.monkey.MonkeyPatcher

We must to path SOURCE_SUFFIXES in the python importlib so we can import source code files with extension other than .py, in this case, namely .conf

patch_source_suffixes()[source]

Patches the SOURCE_SUFFIXES constant in the module importlib._bootstrap_external.

SOURCE_SUFFIXES = ['.py', '.conf']
class toxicbuild.core.utils.changedir(path)[source]

Bases: object

toxicbuild.core.utils.bcrypt_string(src_string, salt=None)[source]
toxicbuild.core.utils.compare_bcrypt_string(original, encrypted)[source]

Compares if a un-encrypted string matches an encrypted one.

Parameters:
  • original – An un-encrypted string.
  • encrypted – An bcrypt encrypted string.
toxicbuild.core.utils.create_random_string(length)[source]
toxicbuild.core.utils.create_validation_string(secret)[source]

Creates a random string that can be used to validate against it. The algorithm is as follows:

Given a secret, a random string is generated, then <secret>-<random-str> are encrypted using bcrypt. Finally <encrypted-str>:<random-str> are base64 encoded.

toxicbuild.core.utils.daemonize(call, cargs, ckwargs, stdout, stderr, workdir, pidfile)[source]

Run a callable as a daemon

Parameters:
  • call – a callable.
  • cargs – args to call.
  • ckwargs – kwargs to call.
  • stdout – daemon’s stdout.
  • stderr – daemon’s stderr.
  • workdir – daemon’s workdir
  • pidfile – pidfile’s path
toxicbuild.core.utils.datetime2string(dt, dtformat='%w %m %d %H:%M:%S %Y %z')[source]

Transforms a datetime object into a formated string.

Parameters:
  • dt – The datetime object.
  • dtformat – The format to use.
toxicbuild.core.utils.exec_cmd(cmd, cwd, timeout=3600, out_fn=None, **envvars)[source]

Executes a shell command. Raises with the command output if return code > 0.

Parameters:
  • cmd – command to run.
  • cwd – Directory to execute the command.
  • timeout – How long we should wait some output. Default is 3600.
  • out_fn – A coroutine that receives each line of the step output. The coroutine signature must be in the form: mycoro(line_index, line).
  • envvars – Environment variables to be used in the command.
toxicbuild.core.utils.format_timedelta(td)[source]

Format a timedelta object to a human-friendly string.

Parameters:dt – A timedelta object.
toxicbuild.core.utils.get_envvars(envvars, use_local_envvars=True)[source]

Returns environment variables to be used in shell. Does the interpolation of values using the current values from the envvar and the values passed as parameters.

toxicbuild.core.utils.inherit_docs(cls)[source]

Inherit docstrings from base classes’ methods. Can be used as a decorator

Parameters:cls – Class that will inherit docstrings from its parents.
toxicbuild.core.utils.interpolate_dict_values(to_return, valued, base)[source]

Interpolates the values of valued with values of base.

Parameters:
  • to_return – A dictionary that will be updated with interpolated values.
  • valued – A dict with values needing interpolation.
  • base – A dict with the base values to use in interpolation.
toxicbuild.core.utils.load_module_from_file(filename)[source]

Load a module from a source file :param filename: full path for file to be loaded.

toxicbuild.core.utils.localtime2utc(localdatetime)[source]

Transforms a local datetime object into a datetime object in utc time.

Parameters:localdatetime – A datetime object.
toxicbuild.core.utils.log(msg, level='info')[source]
toxicbuild.core.utils.match_string(smatch, filters)[source]

Checks if a string match agains a list of filters containing wildcards.

Parameters:
  • smatch – String to test against the filters
  • filters – Filter to match a string.
toxicbuild.core.utils.now()[source]

Returns the localtime with timezone info.

toxicbuild.core.utils.read_file(filename)[source]

Reads the contents of a file asynchronously.

Parameters:filename – The path of the file.
toxicbuild.core.utils.read_stream(reader)[source]

Reads the input stream. First reads the bytes until the first “n”. These first bytes are the length of the full message.

Parameters:reader – An instance of asyncio.StreamReader
toxicbuild.core.utils.run_in_thread(fn, *args, **kwargs)[source]

Runs a callable in a background thread.

Parameters:
  • fn – A callable to be executed in a thread.
  • args – Positional arguments to fn
  • kwargs – Named arguments to fn

Usage

r = yield from run_in_thread(call, 1, bla='a')
toxicbuild.core.utils.set_loglevel(loglevel)[source]
toxicbuild.core.utils.set_tzinfo(dt, tzoff)[source]

Sets a timezone info to a datetime object.

Parameters:dt – A datetime object.
Para tzoff:The timezone offset from utc in seconds
toxicbuild.core.utils.string2datetime(dtstr, dtformat='%w %m %d %H:%M:%S %Y %z')[source]

Transforns a string into a datetime object acording to dtformat.

Parameters:
  • dtstr – The string containing the formated date.
  • dtformat – The format of the formated date.
toxicbuild.core.utils.utc2localtime(utcdatetime)[source]

Transforms a utc datetime object into a datetime object in local time.

Parameters:utcdatetime – A datetime object
toxicbuild.core.utils.validate_string(b64_str, secret)[source]

Validates a string created with create_validattion_string().

Given a base64 string the validation is as follows:

First decodes the base64 string in <encrypted-string>:<random-str> then bcrypt-compare <secret>-<random-str> with <encrypted-string>

toxicbuild.core.utils.write_stream(writer, data)[source]

Writes data to output. Encodes data to utf-8 and prepend the lenth of the data before sending it.

Parameters:
  • writer – An instance of asyncio.StreamWriter
  • data – String data to be sent.

toxicbuild.core.vcs module

class toxicbuild.core.vcs.Git(workdir)[source]

Bases: toxicbuild.core.vcs.VCS

An interface to git version control system

add_remote(remote_url, remote_name)[source]

Adds a new remote to the repository.

Parameters:
  • remote_url – The url of the remote repository.
  • remote_name – The name of the remote.
branch_exists(branch_name)[source]

Checks if a local branch exists.

Parameters:branch_name – The name of the branch to check.
checkout(named_tree)[source]

Checkout to named_tree :param named_tree: A commit, branch, tag…

clone(url)[source]

Clones a repository into self.workdir :param url: repository url

create_local_branch(branch_name, base_name)[source]

Creates a branch new in the local repository

Parameters:
  • branch_name – The name for the new branch
  • base_name – The name of the base branch.
delete_local_branch(branch_name)[source]

Deletes a local branch.

Parameters:branch_name – The name of the branch to be deleted.
fetch()[source]

Fetch changes from remote repository

get_remote(remote_name='origin')[source]

Returns the remote url used in the repo.

Parameters:remote_name – The name of the remote url to change.
get_remote_branches()[source]

Returns a list of the remote branches available.

get_revisions(since=None, branches=None)[source]

Returns the revisions for branches since since.

Parameters:
  • since – dictionary in the format: {branch_name: since_date}. since is a datetime object.
  • branches – A list of branches to look for new revisions. If branches is None all remote branches will be used. You can use wildcards in branches to filter the remote branches.
get_revisions_for_branch(branch, since=None)[source]

Returns the revisions for branch since since. If since is None, all revisions will be returned.

Parameters:
  • branch – branch name
  • since – datetime
has_changes()[source]

Informs if there are new revisions in the repository

coroutine import_external_branch(external_url, external_name, external_branch, into)[source]

Imports a branch from an external (not the origin one) repository into a local branch.

Parameters:
  • external_url – The url of the external repository.
  • external_name – Name to idenfity the remote url.
  • external_branch – The name of the branch in the external repo.
  • into – The name of the local branch.
pull(branch_name, remote_name='origin')[source]

Pull changes from branch_name on remote repo.

Parameters:
  • branch_name – A branch name, like ‘master’.
  • remote_name – The remote repository to push from.
set_remote(url, remote_name='origin')[source]

Sets the remote url of the repository.

Parameters:
  • url – The new remote url.
  • remote_name – The name of the remote url to change.
try_set_remote(url, remote_name='origin')[source]

Sets the remote url if the remote is not equal as url.

Parameters:
  • url – The new url for the remote.
  • remote_name – The name of the remote url to change.
update_submodule()[source]
date_format = '%a %b %d %H:%M:%S %Y'
vcsbin = 'git'
class toxicbuild.core.vcs.VCS(workdir)[source]

Bases: toxicbuild.core.utils.LoggerMixin

Generic inteface to a vcs (clone, fetch, get revisions etc…).

add_remote(remote_url, remote_name)[source]

Adds a new remote to the repository.

Parameters:
  • remote_url – The url of the remote repository.
  • remote_name – The name of the remote.
classmethod branch_exists(branch_name)[source]

Checks if a local branch exists.

Parameters:branch_name – The name of the branch to check.
checkout(named_tree)[source]

Checkout to named_tree :param named_tree: A commit, branch, tag…

clone(url)[source]

Clones a repository into self.workdir :param url: repository url

create_local_branch(branch_name, base_name)[source]

Creates a branch new in the local repository

Parameters:
  • branch_name – The name for the new branch
  • base_name – The name of the base branch.
delete_local_branch(branch_name)[source]

Deletes a local branch.

Parameters:branch_name – The name of the branch to be deleted.
exec_cmd(cmd, cwd=None)[source]

Executes a shell command. If cwd is None self.workdir will be used.

Parameters:cwd – Directory where the command will be executed.
fetch()[source]

Fetch changes from remote repository

get_remote(remote_name)[source]

Returns the remote url used in the repo.

Parameters:remote_name – The name of the remote url to change.
get_remote_branches()[source]

Returns a list of the remote branches available.

get_revisions(since=None, branches=None)[source]

Returns the revisions for branches since since.

Parameters:
  • since – dictionary in the format: {branch_name: since_date}. since is a datetime object.
  • branches – A list of branches to look for new revisions. If branches is None all remote branches will be used. You can use wildcards in branches to filter the remote branches.
get_revisions_for_branch(branch, since=None)[source]

Returns the revisions for branch since since. If since is None, all revisions will be returned.

Parameters:
  • branch – branch name
  • since – datetime
has_changes()[source]

Informs if there are new revisions in the repository

import_external_branch(external_url, external_name, external_branch, into)[source]

Imports a branch from an external (not the origin one) repository into a local branch.

Parameters:
  • external_url – The url of the external repository.
  • external_name – Name to idenfity the remote url.
  • external_branch – The name of the branch in the external repo.
  • into – The name of the local branch.
pull(branch_name, remote_name='origin')[source]

Pull changes from branch_name on remote repo.

Parameters:
  • branch_name – A branch name, like ‘master’.
  • remote_name – The remote repository to push from.
set_remote(url, remote_name)[source]

Sets the remote url of the repository.

Parameters:
  • url – The new remote url.
  • remote_name – The name of the remote url to change.
try_set_remote(url, remote_name)[source]

Sets the remote url if the remote is not equal as url.

Parameters:
  • url – The new url for the remote.
  • remote_name – The name of the remote url to change.
workdir_exists()[source]

Informs if the workdir for this vcs exists

vcsbin = None
toxicbuild.core.vcs.get_vcs(vcs_type)[source]

Retuns a subclass of toxicbuild.core.vcs.VCS for vcs_type

Module contents