2. Common

cloudmesh-common contains a number of useful methods that you can reuse to develop your code. They avoid reimplementation and duplication among the different contributors. Please use these methods instead of reimplementing them.

2.1. DEBUG

2.2. Variable

2.3. Util

Useful utility functions

cloudmesh.common.util.HEADING(txt=None, c='#')

Prints a message to stdout with #### surrounding it. This is useful for nosetests to better distinguish them.

Parameters:

• c – uses the given char to wrap the header
• txt (string) – a text message to be printed

cloudmesh.common.util.auto_create_requirements(requirements)

creates a requirement.txt file form the requirements in the list. If the file exists, it get changed only if the requirements in the list are different from the existing file

Parameters:requirements – the requirements in a list

cloudmesh.common.util.auto_create_version(class_name, version, filename='__init__.py')

creates a version number in the __init__.py file. it can be accessed with __version__ :param class_name: :param version: :param filename: :return:

cloudmesh.common.util.backup_name(filename)

Parameters:filename (string) – given a filename creates a backup name of the form filename.bak.1. If the filename already exists the number will be increased as much as needed so the file does not exist in the given location. The filename can consists a path and is expanded with ~ and environment variables. Return type:string

cloudmesh.common.util.banner(txt=None, c='#', debug=True, label=None, color=None)

prints a banner of the form with a frame of # around the txt:

############################
# txt
############################

Parameters:

• color – prints in the given color
• label – adds a label
• debug – prints only if debug is true
• txt (string) – a text message to be printed
• c (character) – the character used instead of c

cloudmesh.common.util.convert_from_unicode(data)

converts unicode data to a string :param data: the data to convert :return:

cloudmesh.common.util.copy_files(files_glob, source_dir, dest_dir)

Parameters:

• files_glob – *.yaml
• source_dir – source directiry
• dest_dir – destination directory

Returns:

cloudmesh.common.util.exponential_backoff(fn, sleeptime_s_max=1800)

Calls fn until it returns True, with an exponentially increasing wait time between calls

cloudmesh.common.util.generate_password(length=8, lower=True, upper=True, number=True)

generates a simple password. We should not really use this in production. :param length: the length of the password :param lower: True of lower case characters are allowed :param upper: True if upper case characters are allowed :param number: True if numbers are allowed :return:

cloudmesh.common.util.grep(pattern, filename)

Very simple grep that returns the first matching line in a file. String matching only, does not do REs as currently implemented.

cloudmesh.common.util.path_expand(text)

returns a string with expanded variable.

Parameters:

• text – the path to be expanded, which can include ~ and environment $ variables
• text – string

cloudmesh.common.util.readfile(filename)

returns the content of a file :param filename: the filename :return:

cloudmesh.common.util.search(lines, pattern)

return all lines that match the pattern #TODO: we need an example

Parameters:

• lines –
• pattern –

Returns:

cloudmesh.common.util.str_banner(txt=None, c='#', debug=True)

prints a banner of the form with a frame of # around the txt:

############################
# txt
############################

Parameters:

• debug (boolean) – return “” if not in debug
• txt (string) – a text message to be printed
• c (character) – the character used instead of c

cloudmesh.common.util.tempdir(*args, **kwargs)

A contextmanager to work in an auto-removed temporary directory

Arguments are passed through to tempfile.mkdtemp

example:

>>> with tempdir() as path:
...    pass

cloudmesh.common.util.writefile(filename, content)

writes the content into the file :param filename: the filename :param content: teh content :return:

cloudmesh.common.util.yn_choice(message, default='y', tries=None)

asks for a yes/no question.

Parameters:

• tries – the number of tries
• message – the message containing the question
• default – the default answer

2.4. Dotdict

A convenient dot dict class

a = dotdict({“argument”: “value”})

print (a.argument)

class cloudmesh.common.dotdict.dotdict

dot.notation access to dictionary attributes

2.5. Locations

class that specifies where we read the cloudmesh.yaml file from

cloudmesh.common.locations.config_dir_setup(filename)

sets the config file and makes sure the directory exists if it has not yet been created. :param filename: :return:

cloudmesh.common.locations.config_file(filename)

The location of the config file: ~/.cloudmesh/filename. ~ will be expanded :param filename: the filename

cloudmesh.common.locations.config_file_prefix()

The prefix of the configuration file location

cloudmesh.common.locations.config_file_raw(filename)

The location of the config file: ~/.cloudmesh/filename. ~ will NOT be expanded :param filename: the filename

2.6. Parameter

2.7. FlatDict

class cloudmesh.common.FlatDict.FlatDict(d)

A data structure to manage a flattened dict. It is initialized by passing the dict at time of initialization.

keys() → a set-like object providing a view on D's keys

values() → an object providing a view on D's values

cloudmesh.common.FlatDict.flatten(d, parent_key='', sep='__')

flattens the dict into a one dimensional dictionary

Parameters:

• d – multidimensional dict
• parent_key – replaces from the parent key
• sep – the separation character used when fattening. the default is __

Returns:

the flattened dict

cloudmesh.common.FlatDict.key_prefix_replace(d, prefix, new_prefix='')

replaces the list of prefix in keys of a flattened dict

Parameters:

• d – the flattened dict
• prefix (list of str) – a list of prefixes that are replaced with a new prefix. Typically this will be “”
• new_prefix – The new prefix. By default it is set to “”

Returns:

the dict with the keys replaced as specified

2.8. Printer

Convenient methods and classes to print tables.

class cloudmesh.common.Printer.Printer

A simple Printer class with convenient methods to print dictionary, tables, csv, lists

classmethod attribute(d, header=None, order=None, sort_keys=True, output='table')

prints a attribute/key value table :param d: A a dict with dicts of the same type.

Each key will be a column

Parameters:

• order – The order in which the columns are printed. The order is specified by the key names of the dict.
• header (A list of string) – The Header of each of the columns
• sort_keys (string or a tuple of string (for sorting with multiple columns)) – Key(s) of the dict to be used for sorting. This specify the column(s) in the table for sorting.
• output – the output format table, csv, dict, json

classmethod csv(d, order=None, header=None, sort_keys=True)

prints a table in csv format

Parameters:d (dict) – A a dict with dicts of the same type.

:param order:The order in which the columns are printed.

The order is specified by the key names of the dict.

Parameters:

• header (list or tuple of field names) – The Header of each of the columns
• sort_keys (bool) – TODO: not yet implemented

Returns:

a string representing the table in csv format

classmethod dict(d, order=None, header=None, output='table', sort_keys=True, show_none='')

TODO :param d: A a dict with dicts of the same type. :type d: dict :param order:The order in which the columns are printed.

The order is specified by the key names of the dict.

Parameters:

• header (list or tuple of field names) – The Header of each of the columns
• output (string) – type of output (table, csv, json, yaml or dict)
• sort_keys (bool) –
• show_none (bool) – prints None if True for None values otherwise “”

Returns:

classmethod dict_table(d, order=None, header=None, sort_keys=True, show_none='', max_width=40)

prints a pretty table from an dict of dicts :param d: A a dict with dicts of the same type.

Each key will be a column

Parameters:

• order – The order in which the columns are printed. The order is specified by the key names of the dict.
• header (A list of string) – The Header of each of the columns
• sort_keys (string or a tuple of string (for sorting with multiple columns)) – Key(s) of the dict to be used for sorting. This specify the column(s) in the table for sorting.
• show_none (bool) – prints None if True for None values otherwise “”
• max_width (int) – maximum width for a cell

classmethod flatwrite(table, order=None, header=None, output='table', sort_keys=True, show_none='', sep='.')

writes the information given in the table :param table: the table of values :param order: the order of the columns :param header: the header for the columns :param output: the format (default is table, values are raw, csv, json, yaml, dict :param sort_keys: if true the table is sorted :param show_none: passed along to the list or dict printer :param sep: uses sep as the separator for csv printer :return:

classmethod list(l, order=None, header=None, output='table', sort_keys=True, show_none='')

Parameters:

• l – l is a list not a dict
• order –
• header –
• output –
• sort_keys –
• show_none –

Returns:

classmethod print_list(l, output='table')

prints a list :param l: the list :param output: the output, default is a table :return:

classmethod row_table(d, order=None, labels=None)

prints a pretty table from data in the dict. :param d: A dict to be printed :param order: The order in which the columns are printed.

The order is specified by the key names of the dict.

Parameters:labels – The array of labels for the column

classmethod write(table, order=None, header=None, output='table', sort_keys=True, show_none='')

writes the information given in the table :param table: the table of values :param order: the order of the columns :param header: the header for the columns :param output: the format (default is table, values are raw, csv, json, yaml, dict :param sort_keys: if true the table is sorted :param show_none: passed along to the list or dict printer :return:

2.9. Stopwatch

Class for starting and stopping named timers.

This class is based on a similar java class in cyberaide, and java cog kit.

class cloudmesh.common.StopWatch.StopWatch

A class to measure times between events.

classmethod benchmark(sysinfo=True)

prints out all timers in a convenient benchmark tabble :return: :rtype:

classmethod clear()

clear start and end timer_start

classmethod get(name)

returns the time of the timer.

Parameters:name (string) – the name of the timer Return type:the elapsed time

classmethod keys()

returns the names of the timers

classmethod print(*args)

prints a timer. The first argument is the label if it exists, the last is the timer :param args: label, name :return:

classmethod start(name)

starts a timer with the given name.

Parameters:name (string) – the name of the timer

classmethod stop(name)

stops the timer with a given name.

Parameters:name (string) – the name of the timer

2.10. Console

Printing messages in a console

class cloudmesh.common.console.Console

A simple way to print in a console terminal in color. Instead of using simply the print statement you can use special methods to indicate warnings, errors, ok and regular messages.

Example Usage:

Console.warning("Warning")
Console.error("Error")
Console.info("Info")
Console.msg("msg")
Console.ok("Success")

One can switch the color mode off with:

Console.color = False
Console.error("Error")

The color will be switched on by default.

static TODO(message, prefix=True, traceflag=True)

prints an TODO message :param message: the message :param prefix: if set to true it prints TODO: as prefix :param traceflag: if true the stack trace is retrieved and printed :return:

static cprint(color, prefix, message)

prints a message in a given color :param color: the color as defined in the theme :param prefix: the prefix (a string) :param message: the message :return:

static debug_msg(message)

print a debug message :param message: the message :return:

classmethod error(message, prefix=True, traceflag=False)

prints an error message :param message: the message :param prefix: a prefix for the message :param traceflag: if true the stack trace is retrieved and printed :return:

static get(name)

returns the default theme for printing console messages :param name: the name of the theme :return:

static info(message)

prints an informational message :param message: the message :return:

static msg(*message)

prints a message :param message: the message to print :return:

static ok(message)

prints an ok message :param message: the message< :return:

classmethod set_debug(on=True)

sets debugging on or of :param on: if on debugging is set :return:

static set_theme(color=True)

defines if the console messages are printed in color :param color: if True its printed in color :return:

static txt_msg(message, width=79)

prints a message to the screen :param message: the message to print :param width: teh width of the line :return:

static warning(message)

prints a warning :param message: the message :return:

cloudmesh.common.console.indent(text, indent=2, width=128)

indents the given text by the indent specified and wrapping to the given width

Parameters:

• text – the text to print
• indent – indent characters
• width – the width of the text

Returns:

2.11. Logger

simple logging convenience framework

cloudmesh.common.logger.LOGGER(filename)

creates a logger with the given name.

You can use it as follows:

log = cloudmesh.common.LOGGER(__file__)
log.error("this is an error")
log.info("this is an info")
log.warning("this is a warning")

cloudmesh.common.logger.LOGGING_OFF(log)

Switches logging off :param log: the logger for which we switch logging off

cloudmesh.common.logger.LOGGING_ON(log)

Switches logging on :param log: the logger for which we switch logging on

2.12. Error

A simple framework to handle error messages

class cloudmesh.common.error.Error

A class to print error messages

classmethod debug(msg)

prints a debug message. :param msg: the message :return:

classmethod exit(msg)

call a system exit :param msg: :return:

classmethod info(msg)

prints an info msg. :param msg: the message :return:

classmethod msg(error=None, debug=True, trace=True)

prints the error message :param error: the error message :param debug: only prints it if debug is set to true :param trace: if true prints the trace :return:

classmethod traceback(error=None, debug=True, trace=True)

prints the trace :param error: a message preceding the trace :param debug: prints it if debug is set to true :param trace: :return:

classmethod warning(msg)

prints a warning message. :param msg: :return:

2.13. Shell

A convenient method to execute shell commands and return their output. Note: that this method requires that the command be completely execute before the output is returned. FOr many activities in cloudmesh this is sufficient.

class cloudmesh.common.Shell.Shell

The shell class allowing us to conveniently access many operating system commands. TODO: This works well on Linux and OSX, but has not been tested much on Windows

classmethod VBoxManage(*args)

executes VboxManage with the given arguments :param args: :return:

classmethod bash(*args)

executes bash with the given arguments :param args: :return:

classmethod blockdiag(*args)

executes blockdiag with the given arguments :param args: :return:

classmethod brew(*args)

executes bash with the given arguments :param args: :return:

classmethod cat(*args)

executes cat with the given arguments :param args: :return:

classmethod check_output(*args, **kwargs)

Thin wrapper around subprocess.check_output()

classmethod check_python()

checks if the python version is supported :return: True if it is supported

classmethod cm(*args)

executes cm with the given arguments :param args: :return:

command = {'darwin': {}, 'linux': {}, 'windows': {}}

TODO

how do we now define dynamically functions based on a list that we want to support

what we want is where args are multiple unlimited parameters to the function

def f(args…):

name = get the name from f a = list of args…

cls.execute(cmd, arguments=a, capture=True, verbose=False)

commands = [‘ps’, ‘ls’, ….. ] for c in commands:

generate this command and add to this class dynamically

or do something more simple

ls = cls.execute(‘cmd’, args…)

classmethod command_exists(name)

returns True if the command exists :param name: :return:

classmethod dialog(*args)

executes dialof with the given arguments :param args: :return:

classmethod execute(cmd, arguments='', shell=False, cwd=None, traceflag=True, witherror=True)

Run Shell command

Parameters:

• witherror – if set to False the error will not be printed
• traceflag – if set to true the trace is printed in case of an error
• cwd – the current working directory in whcih the command is supposed to be executed.
• shell – if set to true the subprocess is called as part of a shell
• cmd – command to run
• arguments – we do not know yet

Returns:

classmethod fgrep(*args)

executes fgrep with the given arguments :param args: :return:

classmethod find_cygwin_executables()

find the executables in cygwin

classmethod find_lines_with(lines, what)

returns all lines that contain what :param lines: :param what: :return:

classmethod get_python()

returns the python and pip version :return: python version, pip version

classmethod git(*args)

executes git with the given arguments :param args: :return:

classmethod grep(*args)

executes grep with the given arguments :param args: :return:

classmethod head(*args)

executes head with the given arguments :param args: :return:

classmethod keystone(*args)

executes keystone with the given arguments :param args: :return:

classmethod kill(*args)

executes kill with the given arguments :param args: :return:

classmethod ls(*args)

executes ls with the given arguments :param args: :return:

classmethod mkdir(directory)

creates a directory with all its parents in ots name :param directory: the path of the directory :return:

classmethod mongod(*args)

executes mongod with the given arguments :param args: :return:

classmethod nosetests(*args)

executes nosetests with the given arguments :param args: :return:

classmethod nova(*args)

executes nova with the given arguments :param args: :return:

classmethod operating_system()

the name of the os :return: the name of the os

classmethod pandoc(*args)

executes vagrant with the given arguments :param args: :return:

classmethod ping(host=None, count=1)

execute ping :param host: the host to ping :param count: the number of pings :return:

classmethod pip(*args)

executes pip with the given arguments :param args: :return:

classmethod ps(*args)

executes ps with the given arguments :param args: :return:

classmethod pwd(*args)

executes pwd with the given arguments :param args: :return:

classmethod rackdiag(*args)

executes rackdiag with the given arguments :param args: :return:

classmethod remove_line_with(lines, what)

returns all lines that do not contain what :param lines: :param what: :return:

classmethod rm(*args)

executes rm with the given arguments :param args: :return:

classmethod rsync(*args)

executes rsync with the given arguments :param args: :return:

classmethod scp(*args)

executes scp with the given arguments :param args: :return:

classmethod sh(*args)

executes sh with the given arguments :param args: :return:

classmethod sort(*args)

executes sort with the given arguments :param args: :return:

classmethod ssh(*args)

executes ssh with the given arguments :param args: :return:

classmethod sudo(*args)

executes sudo with the given arguments :param args: :return:

classmethod tail(*args)

executes tail with the given arguments :param args: :return:

classmethod terminal_type()

returns darwin, cygwin, cmd, or linux

unzip(source_filename, dest_dir)

unzips a file into the destination directory :param source_filename: the source :param dest_dir: the destination directory :return:

classmethod vagrant(*args)

executes vagrant with the given arguments :param args: :return:

classmethod which(command)

returns the path of the command with which :param command: teh command :return: the path

class cloudmesh.common.Shell.Subprocess(cmd, cwd=None, stderr=-1, stdout=-1, env=None)

Executes a command. This class should not be directly used, but instead you should use Shell.

exception cloudmesh.common.Shell.SubprocessError(cmd, returncode, stderr, stdout)

Manages the formatting of the error and stdout. THis command should not be directly called. Instead use SHell

cloudmesh.common.Shell.main()

a test that should actually be added into a nosetest :return:

2.14. Run

2.15. DB

class cloudmesh.db.strdb.YamlDB(path)

A YAML-backed Key-Value database to store strings

clear()

Truncate the database

close()

This is a NoOP for backwards compatibility

2.16. SSH

authorized key management.

class cloudmesh.common.ssh.authorized_keys.AuthorizedKeys

Class to manage authorized keys.

add(pubkey)

add a public key. :param pubkey: the filename to the public key :return:

classmethod load(path)

load the keys from a path

Parameters:path – the filename (path) in which we find the keys Returns:

remove(pubkey)

Removes the public key TODO: this method is not implemented :param pubkey: the filename of the public key :return:

cloudmesh.common.ssh.authorized_keys.get_fingerprint_from_public_key(pubkey)

Generate the fingerprint of a public key

Parameters:pubkey (str) – the value of the public key Returns:fingerprint Return type:str

Managing ~/.ssh/config

class cloudmesh.common.ssh.ssh_config.ssh_config(filename=None)

Managing the config in .ssh

execute(name, command)

execute the command on the named host :param name: the name of the host in config :param command: the command to be executed :return:

generate(key='india', host='india.futuresystems.org', username=None, force=False, verbose=False)

adds a host to the config file with given parameters. #TODO: make sure this is better documented :param key: the key :param host: the host :param username: the username :param force: not used :param verbose: prints debug messages :return:

list()

list the hosts in the config file :return:

load()

list the hosts defined in the ssh config file

local(command)

execute the command on the localhost :param command: the command to execute :return:

login(name)

login to the host defines in .ssh/config by name :param name: the name of the host as defined in the config file :return:

names()

The names defined in ~/.ssh/config :return: the names

status()

executes a test with the given ssh config if a login is possible.

TODO: not yet implemented

username(host)

returns the username for a given host in the config file :param host: the hostname :return: the username