frkl package

Submodules

frkl.callbacks module

class frkl.callbacks.ExtendResultCallback(**init_params)

Bases: frkl.callbacks.FrklCallback

Simple callback, extends an internal list with the processing results.

callback(process_result)

result()

class frkl.callbacks.FrklCallback(**init_params)

Bases: object

A class to slurp up configurations from the last element of a processor chain.

Since those processers might return Generators, it’s handy to deal with the results of a config processing manually, callbacks seemed like a good way to do it.

callback(item)

Adds a new item to the callback class.

Parameters:item (object) – the newly processed config

finished()

Optional method that can be overwritten if the callback needs to know when the processing has finished.

static init(init_file, configs)

Creates a collector object with associated Frkl and processor chain.

The init file needs to be yaml, with the ‘collector’ key being a registered collector plugin, and the ‘processor_chain’ key being a list of registered ConfigProcessor objects.

Parameters:

• init_file – the path to the init file
• configs – the configuration item(s) for this processor

Returns:

the collector item

Return type:

FrklCallback

result()

Returns a meaningful representation of all added configs so far.

Ideally this is a string representation of the (current) state of the callback, since it might be used by 3rd party tools for debugging purposes, or as a method to show state in the absence of knowledge of the type of FrklCallback that is used.

Returns:the current state of the callback Return type:object

started()

Optional method that can be overwritten if the callback needs to know when the processing has started.

class frkl.callbacks.FrklFactoryCallback(**init_params)

Bases: frkl.callbacks.FrklCallback

Helper callback method, creates a new Frkl object by processing a list of processor init dicts.

callback(item)

result()

class frkl.callbacks.MergeDictResultCallback(**init_params)

Bases: frkl.callbacks.FrklCallback

Simple callback, merges all configs to one internal dict.

callback(process_result)

get_dict_detail(target_dict, detail_path)

result()

set_dict_detail(target_dict, detail_path, new_value)

class frkl.callbacks.MergeResultCallback(**init_params)

Bases: frkl.callbacks.FrklCallback

Simple callback, just appends all configs to an internal list.

callback(process_result)

result()

class frkl.callbacks.SetResultCallback(**init_params)

Bases: frkl.callbacks.FrklCallback

Simple callback to create a set out of a list.

callback(process_result)

result()

frkl.callbacks.load_collector(name, **init_params)

Loading a collector extension.

Parameters:

• name (str) – the registered name of the collector
• init_params (dict) – the parameters to initialize the extension object

Returns:

the extension collector

Return type:

FrklCallback

frkl.chains module

frkl.chains.load_templated_string_from_url_chain(repl_dict, create_python_object=False, use_environment_vars=False, use_context=False, delimiter_profile={'block_end_string': '%}', 'block_start_string': '{%', 'variable_end_string': '}}', 'variable_start_string': '{{'})

Assemles a chain to load a file (local or remote) and replace templated values in it’s content.

Args:

template_values (dict): a dictionary containing the values to replace template strings with create_python_object (bool): whether to return the string (False), or try to convert into python object (True) use_environment_vars (bool): whether to also use current environment variables in the replacement dict. If True, it’ll stored under the key ‘LOCAL_ENV’, if string, that will be used as key use_context (bool): whether to also use the frkl context in the replacement dict. If True, it’ll stored under the key ‘frkl_vars’, if string, that will be used as key delimiter_profile (dict): the jinja2 delimters

Returns:the result string Return type:str

frkl.cli module

class frkl.cli.Config(configuration_dict=None)

Bases: object

frkl configuration, holds things like aliases and such.

frkl.defaults module

frkl.exceptions module

exception frkl.exceptions.FrklConfigException(message, errors=None)

Bases: exceptions.Exception

exception frkl.exceptions.FrklistConfigException(message)

Bases: frkl.exceptions.FrklistException

exception frkl.exceptions.FrklistException(message)

Bases: exceptions.Exception

Base exception class for nsbl.

frkl.frkl module

class frkl.frkl.Frkl(configs=None, processor_chain=[<frkl.processors.UrlAbbrevProcessor object>, <frkl.processors.EnsureUrlProcessor object>, <frkl.processors.EnsurePythonObjectProcessor object>], max_items=9999)

Bases: object

append_configs(configs)

Appends the provided configuration(s) for this Frkl object.

Parameters:configs (list) – the configurations, will wrapped in a list if not a list or tuple already

process(callback=None)

Kicks off the processing of the configuration urls.

Parameters:callback (FrklCallback) – callback to use for this processing run, defaults to ‘MergeResultCallback’ Returns:the value of the result() method of the callback Return type:object

process_single_config(config, processor_chain, callback, configs_copy, context)

Helper method to be able to recursively call the next processor in the chain.

Parameters:

• config (object) – the current config object
• processor_chain (list) – the list of processor items to use (reduces by one with every recursive run)
• callback (FrklCallback) – the callback that receives any potential results
• configs_copy (list) – list of configs that still need processing, this method might prepend newly processed configs to this
• context (dict) – context object, can be used by processors to investigate current state, history, etc.

set_configs(configs)

Sets the configuration(s) for this Frkl object.

Parameters:configs (list) – the configurations, will wrapped in a list if not a list or tuple already

frkl.frkl.load_object_from_url_or_path(urls)

Simple wrapper to create a list of dictionaries from a local or remote file.

If input is a single url, a single list will be returned. If a list, the result will be a list of lists.

As this uses safe_load with the yaml parser, the dictionary will probably not be in the same order as in the original file.

Parameters:

• urls (list) – a list of paths and/or urls
• safe_load (bool) – whether to use safe load with the yaml parser. This will destroy the order of a dict.

Returns:

a list of dictionaries, representing the content of the input files

Return type:

list

frkl.frkl.load_string_from_url_or_path(urls, template_vars=None, delimiter_profile={'block_end_string': '%}', 'block_start_string': '{%', 'variable_end_string': '}}', 'variable_start_string': '{{'}, use_environment_vars=False, use_context=False, create_python_object=False, safe_load=True)

Simple wrapper to create a list of dictionaries from a local or remote file.

If input is a single url, a single list will be returned. If a list, the result will be a list of lists.

Parameters:

• urls (list) – a list of paths and/or urls
• template_vars (dict) – if not None, the string will be considered a jinja2 template and this dict will be used as replacemnt dict
• delimiter_profile (dict) – the delimiter profile, if templating
• use_environment_vars (bool, str) – whether to also use environment variables when templating
• use_context (bool, str) – whether to also use the frkl context when templating
• create_python_object (bool) – whether to create a python object out of the result string or not

Returns:

a string or python object, representing the content of the input files

Return type:

list

frkl.frkl_factory module

frkl.frkl_factory.factory(bootstrap_configs, frkl_configs=None)

Factory method to easily create a Frkl object using a list of configurations to describe the format of the configs to use later on, as well as (optionally) a list of such configs.

Parameters:

• bootstrap_configs (list) – the configuration to describe the format of the configurations the new Frkl object uses
• frkl_configs (list) – (optional) configurations to init the Frkl object with. this can also be done later using the ‘set_configs’ method

Returns:

a new Frkl object

Return type:

Frkl

frkl.frkl_factory.from_folder(folders)

Creates a Frkl object using a folder path as the only input.

The folder needs to contain one or more files that start with the ‘_’ and end with ‘.yml’, which are used to bootstrap the frkl object by reading them in alphabetical order, and one or more additional files with the ‘yml’ extension, which are then used as input configurations, again in alphabetical order.

Parameters:folders (list) – paths to local folder(s) Returns:the initialized Frkl object Return type:Frkl

frkl.frkl_factory.get_configs(folders)

Looks at a folder and retrieves configs.

The folders need to contain one or more files that start with the ‘_’ and end with ‘.yml’, which are used to bootstrap the frkl object by reading them in alphabetical order, and one or more additional files with the ‘yml’ extension, which are then used as input configurations, again in alphabetical order.

Parameters:folders (list) – paths to one or several local folders Returns:first element of the tuple is a list of bootstrap configurations, 2nd element is a list of actual configs Return type:tuple

frkl.frkl_factory.init(files_or_folders, additional_configs=None, use_strings_as_config=False)

Creates a Frkl object.

Parameters:

• files_or_folders (list) – a list of files or folders or url strings. if the item is a file or url string, it will be used as Frkl bootstrap config, if it is a folder, it is forwarded to the ‘from_folder’ method to get lists of bootstrap and/or config files
• additional_configs (list) – a list of files or url strings, used as configs for the initialized Frlk object
• use_strings_as_config (bool) – whether to use non-folder strings as config files (instead of to initialze the Frkl object, which is default)

Returns:

the object

Return type:

Frkl

frkl.frklist module

class frkl.frklist.DefaultFrklist(tasklist, **kwargs)

Bases: frkl.frklist.Frklist

class frkl.frklist.Frklist(tasklist, context=None, meta=None, vars=None)

Bases: object

create_context(context_params)

expand_and_augment_tasklist(tasklist)

preprocess_tasklist(tasklist, context, meta, vars)

process_meta_properties(meta_dict, tasklist)

render_tasklist(**kwargs)

class frkl.frklist.FrklistContext(base_directory=None, allow_remote=False, allow_untrusted_urls=False, trusted_urls=None, abbrevs=None, environment_paths=None, template_vars=None)

Bases: object

cleanup_environment_paths(environment_paths)

frkl.frklist.GLOBAL_ENV_ID()

frkl.frklist.GLOBAL_TASKLIST_ID()

frkl.frklist.create_tasklist(tasklist, context=None, meta=None, vars=None)

Loading the tasklist class

Parameters:

• name (str) – the registered name of the extension
• init_params (dict) – the parameters to initialize the extension object

Returns:

the extension object

Return type:

DictletFinder

frkl.helpers module

frkl.helpers.content_from_url(abbrev_or_url, update=False, cache_base='/home/docs/.local/share/frkl/download_cache', abbrevs=None)

Downloads a file using a full or abbreviated url and returns it’s content.

Parameters:

• abbrev_or_url (str) – the full or abbreviated url
• update (bool) – whether to ‘force’ update the file if it doesn’t exist
• cache_base (str) – root of cache directory
• abbrevs (dict) – (optional) url abbreviations

Returns:

the content of the downloaded file

Return type:

str

frkl.helpers.dict_from_url(abbrev_or_url, update=False, cache_base='/home/docs/.local/share/frkl/download_cache', abbrevs=None)

Downloads a file using a full or abbreviated url and returns it’s content as a python dict.

Parameters:

• abbrev_or_url (str) – the full or abbreviated url
• update (bool) – whether to ‘force’ update the file if it doesn’t exist
• cache_base (str) – root of cache directory
• abbrevs (dict) – (optional) url abbreviations

Returns:

the content of the downloaded file

Return type:

str

frkl.helpers.download_cached_file(abbrev_or_url, update=False, cache_base='/home/docs/.local/share/frkl/download_cache', abbrevs=None, return_content=False)

Downloads a file using a full or abbreviated url.

Parameters:

• abbrev_or_url (str) – the full or abbreviated url
• update (bool) – whether to ‘force’ update the file if it doesn’t exist
• cache_base (str) – root of cache directory
• abbrevs (dict) – (optional) url abbreviations
• return_content (bool) – whether to return the path to the file (False) or it’s content (True)

Returns:

the path to the downloaded file or it’s content (depending on the ‘return_content’ variable)

Return type:

str

frkl.helpers.get_full_url(abbrev_or_url, abbrevs=None)

Expands input to a full url if abbreviated.

Parameters:abbrev_or_url (str) – a full or abbreviated url

Returns

str: the full url

frkl.processors module

class frkl.processors.ConfigProcessor(**init_params)

Bases: object

Abstract base class for config url/content manipulators.

In order to enable configuration urls and content to be written as quickly and minimal as possible, frkl supports pluggable processors that can manipulate the configuration urls and contents. For example, urls can be abbreviated ‘gh’ -> ‘https://raw.githubusercontent.com/blahblah’.

get_additional_configs()

Returns additional configs if applicable.

This is called before the ‘process’ method.

Returns:other configs to be processed next Return type:list

get_input_format()

Returns the format of the accepted input.

Defaults to ‘STRING’

get_output_format()

Returns the format of the output.

Defaults to the same as the ‘get_input_format’ method.

handles_last_call()

Returns whether this processor wants to be called at the end of a processing run again.

If the preceding processor returns a non-None value, this is ignored and the processor is called anyway.

Returns:whether to call this processor for the ‘special’ last run Return type:bool

new_config()

Can be overwritten to initially compute a new config after it is first set.

The newly (last) added input configuration is stored in the ‘self.current_input_config’ variable.

process()

Processes the config url or content.

Calls the ‘process_current_config’ method internally

Returns:processed config url or content Return type:object

process_current_config()

Processes the config url or content.

Returns:processed config url or content Return type:object

set_current_config(input_config, context)

Sets the current configuration.

Calls the ‘new_config’ method after assigning the new input configuration to the ‘self.current_input_config’ variable.

Parameters:

• input_config (object) – current configuration to be processed
• context (dict) – dict that describes the current context / processing state

class frkl.processors.DictInjectionProcessor(**init_params)

Bases: frkl.processors.ConfigProcessor

A processor to ‘inject’ dictionaries and dictionary values into other dictionaries, according to predefined rules.

get_input_format()

process_current_config()

class frkl.processors.EnsurePythonObjectProcessor(safe_load=True, **kwargs)

Bases: frkl.processors.ConfigProcessor

Makes sure the provided string is either valid yaml (or json – not implemented yet), and converts it into a python object.

Parameters:safe (bool) – whether to use the ‘safe’ type for the yaml parser. This will destroy the order of any source dicts.

get_output_format()

process_current_config()

class frkl.processors.EnsureUrlProcessor(**init_params)

Bases: frkl.processors.ConfigProcessor

Makes sure the provided string is a url, then downloads the target and reads the content.

get_config(config_file_url)

Retrieves the config (if necessary), and returns its content.

Config can be either a path to a local yaml file, an url to a remote yaml file, or a json string.

Parameters:config_file_url (str) – the url/path/json content

process_current_config()

class frkl.processors.FrklProcessor(**init_params)

Bases: frkl.processors.ConfigProcessor

A processor to ‘expand’ python dictionaries using a pre-defined schema.

This is a bit more complicated to explain than I’d like it to be. For that reason, there is an extra page in the docs: link (XXX)

frklize(config, current_vars)

Recursively called function which generates (expands) and yields dictionaries matching certain criteria (containing leaf_node keys, for example).

Parameters:

• config (object) – the input config
• current_vars (dict) – current state of the (overlayed) var cache

get_input_format()

new_config()

process_current_config()

class frkl.processors.IdProcessor(**init_params)

Bases: frkl.processors.ConfigProcessor

Adds an id to every config item.

get_input_format()

process_current_config()

validate_init()

class frkl.processors.Jinja2TemplateProcessor(template_values=None, use_environment_vars=False, use_context=False, delimiter_profile={'block_end_string': '%}', 'block_start_string': '{%', 'variable_end_string': '}}', 'variable_start_string': '{{'}, **init_params)

Bases: frkl.processors.ConfigProcessor

Processor to replace all occurrences of Jinja template strings with values (predefined, or potentially dynamically processed in an earlier step).

Parameters:

• template_values (dict) – a dictionary containing the values to replace template strings with
• use_environment_vars (bool) – whether to also use current environment variables in the replacement dict. If True, it’ll stored under the key ‘LOCAL_ENV’, if string, that will be used as key
• use_context (bool) – whether to also use the frkl context in the replacement dict. If True, it’ll stored under the key ‘frkl_vars’, if string, that will be used as key
• delimiter_profile (dict) – the jinja2 delimters
• **init_params (dict) – additional config for the parent class

get_input_format()

process_current_config()

class frkl.processors.LoadMoreConfigsProcessor(**init_params)

Bases: frkl.processors.ConfigProcessor

Processort to load additional configs from configs.

If an incoming configuration is a list of strings, it’ll interprete it as list of urls and adds it in front of the list of ‘yet-to-process’ configs of this processing run.

Use this with caution, since if this gets a list of string that is not a list of urls, it will still treat it like one and your run will fail.

get_additional_configs()

get_input_format()

get_output_format()

process_current_config()

class frkl.processors.MergeProcessor(**init_params)

Bases: frkl.processors.ConfigProcessor

Gathers all configs and returns a list of all results as single element.

get_input_format()

handles_last_call()

new_config()

process_current_config()

class frkl.processors.ParentPathProcessor(**init_params)

Bases: frkl.processors.ConfigProcessor

get_input_format()

get_output_format()

process_current_config()

class frkl.processors.RegexProcessor(**init_params)

Bases: frkl.processors.ConfigProcessor

Replaces all occurences of regex matches.

Parameters:regexes (dict) – a map of regexes and their replacements

get_input_format()

process_current_config()

class frkl.processors.ToYamlProcessor(**init_params)

Bases: frkl.processors.ConfigProcessor

Takes a python object and returns the string representation.

get_input_format()

get_output_format()

process_current_config()

class frkl.processors.UrlAbbrevProcessor(**init_params)

Bases: frkl.processors.ConfigProcessor

Replaces strings in an input configuration url with its expanded version.

The default constructor without any arguments will create a processor only using the default, inbuilt abbreviations

expand_config(config)

Expands abbreviated configuration urls that start with <token>:.

This is a convenience for the user, as they don’t have to type out long urls if they don’t want to.

To make it easier to remember (and shorter to type) config urls, frkl supports abbreviations which will be replaced before attempting to load a configuration. In addition to inbuild abbreviations, a custom ones can be piped into the Frkl constructor. This needs to be a dictionary of string to string (abbreviation -> full url-part, eg. freckles_configs --> https://raw.githubusercontent.com/makkus/freckles/master/examples, which would enable the user to provide this config url: freckles_config:quickstart.yml) or string to list (abbreviation -> list of tokens to assemble the finished string, which for example for getting raw files from the master branch of a github repo would look like: [“https://raw.githubusercontent.com”, frutils.defaults.URL_PLACEHOLDER, frutils.defaults.URL_PLACEHOLDER, “master] – the input config gh:makkus/freckles/examples/quickstart.yml would result in the same string as in the first example above – this method is a bit more fragile, because the input string can’t have less tokens seperated by ‘/’ than the value list).

Parameters:config (str) – the configuration url/json/etc… Returns:the configuration with all occurances of registered abbreviations replaced Return type:str

get_input_format()

process_current_config()

class frkl.processors.YamlTextSplitProcessor(**init_params)

Bases: frkl.processors.ConfigProcessor

Splits a string if it can find certain keywords at the beginning of a line.

Parameters:keywords (list) – a list of keywords

get_input_format()

handles_last_call()

process_current_config()

frkl.processors.load_extension(name, **init_params)

Loading a processor extension.

Parameters:

• name (str) – the registered name of the extension
• init_params (dict) – the parameters to initialize the extension object

Returns:

the extension object

Return type:

FrklConfig

frkl.utils module

frkl.utils.expand_string_to_git_details(value, default_abbrevs)

frkl.utils.expand_string_to_git_repo(value, default_abbrevs)

frkl.utils.get_url_parents(urls, abbrevs=False, return_list=False)

Helper methods to calculate the parents of the provided urls.

Parameters:

• urls (list) – the list of urls
• abbrevs (bool, dict) – if False, urls won’t be expanded if they are abbreviated, otherwise if a abbrev dict is provided, they will be
• return_list (bool) – whether to return the result as set (False) or list (True)

Module contents

View page source