toxicbuild.core package

Submodules

toxicbuild.core.build_config module

class toxicbuild.core.build_config.APTPluginConfig(conf)

Bases: toxicbuild.core.build_config.BasePluginConfig

get_config()

class toxicbuild.core.build_config.BasePluginConfig(conf)

Bases: object

Plugin configs are meant to be used along with the LanguageConfig. Based in the toxicbuild.yml config we can create the build plugins configuration for a builder.

Note

The build plugins currently live in the slave but that is a very silly idea. In a future relase that’s gonna change.

class toxicbuild.core.build_config.LanguageConfig(conf)

Bases: object

An abstraction to create builders based in a language config in a toxicbuild.yml config file.

DEFAULT_OS = 'debian'

LANGUAGE_PLUGINS = {'python': <class 'toxicbuild.core.build_config.PythonPluginConfig'>}

SYSTEM_PACKAGES_PLUGINS = {'debian': <class 'toxicbuild.core.build_config.APTPluginConfig'>, 'ubuntu': <class 'toxicbuild.core.build_config.APTPluginConfig'>}

builders

class toxicbuild.core.build_config.LanguagePluginConfig(lang_ver, conf)

Bases: toxicbuild.core.build_config.BasePluginConfig

class toxicbuild.core.build_config.PythonPluginConfig(lang_ver, conf)

Bases: toxicbuild.core.build_config.LanguagePluginConfig

get_config()

toxicbuild.core.build_config.get_config(workdir, config_type, config_filename)

Returns the build configuration for a given repository.

Parameters:

• workdir – The directory where config file is located.
• config_type – Which type of configuration we are using, ‘py’ or ‘yaml’.
• config_filename – The filename of the config file.

toxicbuild.core.build_config.get_toxicbuildconf(directory)

Returns the toxicbuild.conf module.

Parameters:directory – Directory to look for toxicbuild.conf

toxicbuild.core.build_config.get_toxicbuildconf_yaml(directory, filename='toxicbuild.yml')

Returns the python objet representing the yaml build configuration.

Parameters:

• directory – The path of the directory containing the config file.
• filename – The actual name of the config file.

toxicbuild.core.build_config.list_builders_from_config(config, branch=None, slave=None, config_type='yml')

Lists builders from a build config

Parameters:

• config – The build configuration.
• branch – The branch for which builders are being listed.
• slave – The slave for which builders are being listed.

toxicbuild.core.client module

This module implements a base client for basic tcp communication, reading and writing json data.

Usage:

host = 'somehost.net'
port = 1234
async with BaseToxicClient(host, port):
    await client.write({'hello': 'world'})
    json_response = await client.get_response()

class toxicbuild.core.client.BaseToxicClient(host, port, loop=None, use_ssl=False, validate_cert=True, **ssl_kw)

Bases: toxicbuild.core.utils.LoggerMixin

Base client for communication with toxicbuild servers.

coroutine connect()

Connects to the server.

Note

This is called by the asynchronous context manager (aka async with)

disconnect()

Disconnects from the server

coroutine get_response()

Reads data from the server and raises and exception in case of error

is_connected()

coroutine read()

Reads data from the server. Expects a json.

coroutine write(data)

Writes data to the server.

Parameters:data – Data to be sent to the server. Will be converted to json and enconded using utf-8.

toxicbuild.core.cmd module

class toxicbuild.core.cmd.ToxicProgram(*args, **kwargs)

Bases: mando.core.Program

Transforms a python function into a command line program. Extends mando’s Program to execute _generate_command() only when execute() is called to enable toxicbuild command hacks.

execute(*args, **kwargs)

Parse the arguments and execute the resulting command.

Parameters:args – The arguments to parse.

toxicbuild.core.conf module

class toxicbuild.core.conf.Settings(envvar, default_filename)

Bases: object

Simple interface to a settings file.

toxicbuild.core.exceptions module

exception toxicbuild.core.exceptions.BadJsonData

Bases: Exception

exception toxicbuild.core.exceptions.ConfigError

Bases: Exception

exception toxicbuild.core.exceptions.ExecCmdError

Bases: Exception

exception toxicbuild.core.exceptions.ImpossibillityError

Bases: Exception

exception toxicbuild.core.exceptions.PluginNotFound

Bases: Exception

exception toxicbuild.core.exceptions.ToxicClientException

Bases: Exception

exception toxicbuild.core.exceptions.VCSError

Bases: Exception

toxicbuild.core.exchange module

class toxicbuild.core.exchange.AmqpConnection(**conn_kwargs)

Bases: toxicbuild.core.utils.LoggerMixin

A connection for a broker. We can have many channels over one connection.

coroutine connect()

Connects to the Rabbitmq server.

coroutine disconnect()

Disconnects from the Rabbitmq server.

class toxicbuild.core.exchange.Consumer(channel, queue, consumer_tag, nowait=False, timeout=0)

Bases: asyncamqp.consumer.Consumer, toxicbuild.core.utils.LoggerMixin

coroutine fetch_message(cancel_on_timeout=True)

class toxicbuild.core.exchange.Exchange(name, connection, exchange_type, durable=False, bind_publisher=True, exclusive_consumer_queue=False)

Bases: toxicbuild.core.utils.LoggerMixin

A simple abstraction for an amqp exchange.

coroutine bind(routing_key, queue_name=None, channel=None)

Binds the queue to the exchange.

Parameters:

• routing_key – Routing key to bind the queue.
• queue_name – The name of the queue to be bound. If not self.queue_name will be used.
• channel – Optional channel to use in the communication. A new one will be used.

coroutine consume(wait_message=True, timeout=0, routing_key='', no_ack=False)

Consumes a message from a Rabbitmq queue.

Parameters:

• wait_message – Should we wait for new messages in the queue?
• timeout – Timeout for waiting messages.
• no_ack – Indicates if we should send a ack response to the server. The ack must be sent by the consumer.

coroutine declare(queue_name=None)

Declares the exchange and queue.

Parameters:queue_name – The name for the queue to be declared. If None, self.queue_name will be used.

coroutine get_queue_size(queue_name=None)

is_declared(queue_name=None)

coroutine publish(message, routing_key='')

Publishes a message to a Rabbitmq exchange

Parameters:

• message – The message that will be published in the exchange. Must be something that can be serialized into a json.
• routing_key – The routing key to pdublish the message.

coroutine queue_delete(queue_name=None)

Deletes a queue. If not queue_name, self.queue_name will be used.

Parameters:queue_name – The name of the queue to be deleted.

coroutine unbind(routing_key, channel=None)

class toxicbuild.core.exchange.JsonAckMessage(*args, **kwargs)

Bases: asyncamqp.consumer.Message

coroutine acknowledge()

coroutine reject(requeue=True)

toxicbuild.core.exchanges module

toxicbuild.core.exchanges.create_exchanges(conn)

toxicbuild.core.exchanges.disconnect_exchanges()

toxicbuild.core.mail module

exception toxicbuild.core.mail.MailSenderNotConnected

Bases: Exception

class toxicbuild.core.mail.MailSender(recipients, **smtp_settings)

Bases: toxicbuild.core.utils.LoggerMixin

Simple mail sender.

To send an email, use the context manager:

recipients = ['me@mail.com', 'other@mail.com']
smtp_settings = {'host': 'some.host', 'port': 123,
                 'mail_from': 'me@myplace.net',
                 'username': 'me', 'password': '1234',
                 'validate_certs': True, 'starttls': False}

async with MailSender(recipients, **smtp_settigs) as sender:
    await sender.send('Subject', 'This is the message body')

coroutine connect()

Connects to a smtp server.

coroutine disconnect()

Closes the connection to the smpt server

coroutine send(subject, message)

Send an email message.

Parameters:

• subject – Email subject.
• message – Email body

toxicbuild.core.plugins module

class toxicbuild.core.plugins.Plugin(*args, **kwargs)

Bases: object

This is a base plugin. Plugins may implement aditional behavior to your builds.

classmethod get_plugin(name)

Returns a Plugin subclass based on its name.

Parameters:name – Plugin’s name.

classmethod list_plugins(plugin_type=None)

Returns a list of Plugin subclasses.

Parameters:plugin_type – the plugin’s type.

name = 'BaseCorePlugin'

no_list = False

class toxicbuild.core.plugins.PluginMeta

Bases: type

toxicbuild.core.protocol module

class toxicbuild.core.protocol.BaseToxicProtocol(loop, connection_lost_cb=None)

Bases: asyncio.streams.StreamReaderProtocol, toxicbuild.core.utils.LoggerMixin

Base protocol for toxicbulid servers

check_data()

Checks if the data is valid, it means, checks if has some data, checks if it is a valid json and checks if it has a action key

client_connected()

Coroutine that handles connections. You must implement this in your sub-classes. When this method is called, self.data, containing a dictionary with the data passed in the request and self.action, a string indicating which action to take are already available.

close_connection()

Closes the connection with the client

connection_lost(exc)

Called once, when the connection is lost.

Parameters:exc – The exception, if some.

connection_made(transport)

Called once, when the client connects.

Parameters:transport – transport for asyncio.StreamReader and asyncio.StreamWriter.

get_json_data()

Returns the json sent by the client.

get_raw_data()

Returns the raw data sent by the client

send_response(code, body)

Send a response to client formated by the (unknown) toxicbuild remote build specs.

Parameters:

• code – code for this message. code == 0 is success and code > 0 is error.
• body – response body. It has to be a serializable object.

encrypted_token = None

toxicbuild.core.requests module

This module implements a simple asynchronous interface for http requests.

Usage:

from toxicbuild.core import requests
response = yield from requests.get('http://google.com/')
print(response.text)

class toxicbuild.core.requests.Response(status, text, headers=None)

Bases: object

Encapsulates a response from a http request

json()

Loads the json in the response text.

toxicbuild.core.requests.delete(url, **kwargs)

Performs a http DELETE request

Parameters:

• url – Request’s url.
• kwargs – Args passed to toxicbuild.core.requests._request().

toxicbuild.core.requests.get(url, **kwargs)

Performs a http GET request

Parameters:

• url – Request’s url.
• kwargs – Args passed to toxicbuild.core.requests._request().

toxicbuild.core.requests.post(url, **kwargs)

Performs a http POST request

Parameters:

• url – Request’s url.
• kwargs – Args passed to toxicbuild.core.requests._request().

toxicbuild.core.requests.put(url, **kwargs)

Performs a http PUT request

Parameters:

• url – Request’s url.
• kwargs – Args passed to toxicbuild.core.requests._request().

toxicbuild.core.utils module

class toxicbuild.core.utils.LoggerMixin

Bases: object

A simple mixin to use log on a class.

log(msg, level='info')

Appends the class name before the log message.

classmethod log_cls(msg, level='info')

class toxicbuild.core.utils.MatchKeysDict

Bases: dict

A dictionary that returns the values matching the keys using toxicbuild.core.utils.match_string().

>>> d = MatchKeysDict()
>>> d['k*'] = 1
>>> d['key']
1
>>> k['keyboard']
1

get(k[, d]) → D[k] if k in D, else d. d defaults to None.

class toxicbuild.core.utils.SettingsPatcher

Bases: mongomotor.monkey.MonkeyPatcher

Patches the settings from pyrocumulus to use the same settings as toxibuild.

patch_pyro_settings(settings)

class toxicbuild.core.utils.SourceSuffixesPatcher

Bases: mongomotor.monkey.MonkeyPatcher

We must to path SOURCE_SUFFIXES in the python importlib so we can import source code files with extension other than .py, in this case, namely .conf

patch_source_suffixes()

Patches the SOURCE_SUFFIXES constant in the module importlib._bootstrap_external.

SOURCE_SUFFIXES = ['.py', '.conf']

class toxicbuild.core.utils.changedir(path)

Bases: object

toxicbuild.core.utils.bcrypt_string(src_string, salt=None)

toxicbuild.core.utils.compare_bcrypt_string(original, encrypted)

Compares if a un-encrypted string matches an encrypted one.

Parameters:

• original – An un-encrypted string.
• encrypted – An bcrypt encrypted string.

toxicbuild.core.utils.create_random_string(length)

toxicbuild.core.utils.create_validation_string(secret)

Creates a random string that can be used to validate against it. The algorithm is as follows:

Given a secret, a random string is generated, then <secret>-<random-str> are encrypted using bcrypt. Finally <encrypted-str>:<random-str> are base64 encoded.

toxicbuild.core.utils.daemonize(call, cargs, ckwargs, stdout, stderr, workdir, pidfile)

Run a callable as a daemon

Parameters:

• call – a callable.
• cargs – args to call.
• ckwargs – kwargs to call.
• stdout – daemon’s stdout.
• stderr – daemon’s stderr.
• workdir – daemon’s workdir
• pidfile – pidfile’s path

toxicbuild.core.utils.datetime2string(dt, dtformat='%w %m %d %H:%M:%S %Y %z')

Transforms a datetime object into a formated string.

Parameters:

• dt – The datetime object.
• dtformat – The format to use.

toxicbuild.core.utils.exec_cmd(cmd, cwd, timeout=3600, out_fn=None, **envvars)

Executes a shell command. Raises with the command output if return code > 0.

Parameters:

• cmd – command to run.
• cwd – Directory to execute the command.
• timeout – How long we should wait some output. Default is 3600.
• out_fn – A coroutine that receives each line of the step output. The coroutine signature must be in the form: mycoro(line_index, line).
• envvars – Environment variables to be used in the command.

toxicbuild.core.utils.format_timedelta(td)

Format a timedelta object to a human-friendly string.

Parameters:dt – A timedelta object.

toxicbuild.core.utils.get_envvars(envvars, use_local_envvars=True)

Returns environment variables to be used in shell. Does the interpolation of values using the current values from the envvar and the values passed as parameters.

toxicbuild.core.utils.inherit_docs(cls)

Inherit docstrings from base classes’ methods. Can be used as a decorator

Parameters:cls – Class that will inherit docstrings from its parents.

toxicbuild.core.utils.interpolate_dict_values(to_return, valued, base)

Interpolates the values of valued with values of base.

Parameters:

• to_return – A dictionary that will be updated with interpolated values.
• valued – A dict with values needing interpolation.
• base – A dict with the base values to use in interpolation.

toxicbuild.core.utils.load_module_from_file(filename)

Load a module from a source file :param filename: full path for file to be loaded.

toxicbuild.core.utils.localtime2utc(localdatetime)

Transforms a local datetime object into a datetime object in utc time.

Parameters:localdatetime – A datetime object.

toxicbuild.core.utils.log(msg, level='info')

toxicbuild.core.utils.match_string(smatch, filters)

Checks if a string match agains a list of filters containing wildcards.

Parameters:

• smatch – String to test against the filters
• filters – Filter to match a string.

toxicbuild.core.utils.now()

Returns the localtime with timezone info.

toxicbuild.core.utils.read_file(filename)

Reads the contents of a file asynchronously.

Parameters:filename – The path of the file.

toxicbuild.core.utils.read_stream(reader)

Reads the input stream. First reads the bytes until the first “n”. These first bytes are the length of the full message.

Parameters:reader – An instance of asyncio.StreamReader

toxicbuild.core.utils.run_in_thread(fn, *args, **kwargs)

Runs a callable in a background thread.

Parameters:

• fn – A callable to be executed in a thread.
• args – Positional arguments to fn
• kwargs – Named arguments to fn

Usage

r = yield from run_in_thread(call, 1, bla='a')

toxicbuild.core.utils.set_loglevel(loglevel)

toxicbuild.core.utils.set_tzinfo(dt, tzoff)

Sets a timezone info to a datetime object.

Parameters:dt – A datetime object. Para tzoff:The timezone offset from utc in seconds

toxicbuild.core.utils.string2datetime(dtstr, dtformat='%w %m %d %H:%M:%S %Y %z')

Transforns a string into a datetime object acording to dtformat.

Parameters:

• dtstr – The string containing the formated date.
• dtformat – The format of the formated date.

toxicbuild.core.utils.utc2localtime(utcdatetime)

Transforms a utc datetime object into a datetime object in local time.

Parameters:utcdatetime – A datetime object

toxicbuild.core.utils.validate_string(b64_str, secret)

Validates a string created with create_validattion_string().

Given a base64 string the validation is as follows:

First decodes the base64 string in <encrypted-string>:<random-str> then bcrypt-compare <secret>-<random-str> with <encrypted-string>

toxicbuild.core.utils.write_stream(writer, data)

Writes data to output. Encodes data to utf-8 and prepend the lenth of the data before sending it.

Parameters:

• writer – An instance of asyncio.StreamWriter
• data – String data to be sent.

toxicbuild.core.vcs module

class toxicbuild.core.vcs.Git(workdir)

Bases: toxicbuild.core.vcs.VCS

An interface to git version control system

add_remote(remote_url, remote_name)

Adds a new remote to the repository.

Parameters:

• remote_url – The url of the remote repository.
• remote_name – The name of the remote.

branch_exists(branch_name)

Checks if a local branch exists.

Parameters:branch_name – The name of the branch to check.

checkout(named_tree)

Checkout to named_tree :param named_tree: A commit, branch, tag…

clone(url)

Clones a repository into self.workdir :param url: repository url

create_local_branch(branch_name, base_name)

Creates a branch new in the local repository

Parameters:

• branch_name – The name for the new branch
• base_name – The name of the base branch.

delete_local_branch(branch_name)

Deletes a local branch.

Parameters:branch_name – The name of the branch to be deleted.

fetch()

Fetch changes from remote repository

get_remote(remote_name='origin')

Returns the remote url used in the repo.

Parameters:remote_name – The name of the remote url to change.

get_remote_branches()

Returns a list of the remote branches available.

get_revisions(since=None, branches=None)

Returns the revisions for branches since since.

Parameters:

• since – dictionary in the format: {branch_name: since_date}. since is a datetime object.
• branches – A list of branches to look for new revisions. If branches is None all remote branches will be used. You can use wildcards in branches to filter the remote branches.

get_revisions_for_branch(branch, since=None)

Returns the revisions for branch since since. If since is None, all revisions will be returned.

Parameters:

• branch – branch name
• since – datetime

has_changes()

Informs if there are new revisions in the repository

coroutine import_external_branch(external_url, external_name, external_branch, into)

Imports a branch from an external (not the origin one) repository into a local branch.

Parameters:

• external_url – The url of the external repository.
• external_name – Name to idenfity the remote url.
• external_branch – The name of the branch in the external repo.
• into – The name of the local branch.

pull(branch_name, remote_name='origin')

Pull changes from branch_name on remote repo.

Parameters:

• branch_name – A branch name, like ‘master’.
• remote_name – The remote repository to push from.

set_remote(url, remote_name='origin')

Sets the remote url of the repository.

Parameters:

• url – The new remote url.
• remote_name – The name of the remote url to change.

try_set_remote(url, remote_name='origin')

Sets the remote url if the remote is not equal as url.

Parameters:

• url – The new url for the remote.
• remote_name – The name of the remote url to change.

update_submodule()

date_format = '%a %b %d %H:%M:%S %Y'

vcsbin = 'git'

class toxicbuild.core.vcs.VCS(workdir)

Bases: toxicbuild.core.utils.LoggerMixin

Generic inteface to a vcs (clone, fetch, get revisions etc…).

add_remote(remote_url, remote_name)

Adds a new remote to the repository.

Parameters:

• remote_url – The url of the remote repository.
• remote_name – The name of the remote.

classmethod branch_exists(branch_name)

Checks if a local branch exists.

Parameters:branch_name – The name of the branch to check.

checkout(named_tree)

Checkout to named_tree :param named_tree: A commit, branch, tag…

clone(url)

Clones a repository into self.workdir :param url: repository url

create_local_branch(branch_name, base_name)

Creates a branch new in the local repository

Parameters:

• branch_name – The name for the new branch
• base_name – The name of the base branch.

delete_local_branch(branch_name)

Deletes a local branch.

Parameters:branch_name – The name of the branch to be deleted.

exec_cmd(cmd, cwd=None)

Executes a shell command. If cwd is None self.workdir will be used.

Parameters:cwd – Directory where the command will be executed.

fetch()

Fetch changes from remote repository

get_remote(remote_name)

Returns the remote url used in the repo.

Parameters:remote_name – The name of the remote url to change.

get_remote_branches()

Returns a list of the remote branches available.

get_revisions(since=None, branches=None)

Returns the revisions for branches since since.

Parameters:

• since – dictionary in the format: {branch_name: since_date}. since is a datetime object.
• branches – A list of branches to look for new revisions. If branches is None all remote branches will be used. You can use wildcards in branches to filter the remote branches.

get_revisions_for_branch(branch, since=None)

Returns the revisions for branch since since. If since is None, all revisions will be returned.

Parameters:

• branch – branch name
• since – datetime

has_changes()

Informs if there are new revisions in the repository

import_external_branch(external_url, external_name, external_branch, into)

Imports a branch from an external (not the origin one) repository into a local branch.

Parameters:

• external_url – The url of the external repository.
• external_name – Name to idenfity the remote url.
• external_branch – The name of the branch in the external repo.
• into – The name of the local branch.

pull(branch_name, remote_name='origin')

Pull changes from branch_name on remote repo.

Parameters:

• branch_name – A branch name, like ‘master’.
• remote_name – The remote repository to push from.

set_remote(url, remote_name)

Sets the remote url of the repository.

Parameters:

• url – The new remote url.
• remote_name – The name of the remote url to change.

try_set_remote(url, remote_name)

Sets the remote url if the remote is not equal as url.

Parameters:

• url – The new url for the remote.
• remote_name – The name of the remote url to change.

workdir_exists()

Informs if the workdir for this vcs exists

vcsbin = None

toxicbuild.core.vcs.get_vcs(vcs_type)

Retuns a subclass of toxicbuild.core.vcs.VCS for vcs_type

Module contents