Internal API

pipsalabim

pipsalabim is just black magic.

Pip Sala Bim is a package that studies the codebase of your project in search for internal and external imports. It then discards the imports that are satisfied with internal code or with the standard library and finally searches the PyPIContents index to list which packages satisfy your imports.

pipsalabim.cli module

pipsalabim.cli is a module for handling the command line interface.

This module handles the commands for using Pip Sala Bim. It also parses parameters, show help, version and controls the logging level.

pipsalabim.cli.commandline(argv=None)[source]

Configure ArgumentParser to accept custom arguments and commands.

Parameters:argv

a list of commandline arguments like sys.argv. For example:

['-v', '--loglevel', 'INFO']
Returns:a Namespace object.

New in version 0.1.0.

pipsalabim.cli.main(argv=None)[source]

Handle arguments and commands.

Parameters:argv

a list of commandline arguments like sys.argv. For example:

['-v', '--loglevel', 'INFO']
Returns:an exit status.

New in version 0.1.0.

pipsalabim.api package

pipsalabim.api is a package containing all the cli commands.

This package contains each command of Pip Sala Bim organized in modules.

pipsalabim.api.report module

pipsalabim.api.report is a module implementing the report command.

This module contains the logic to examine your source code, extract internal and external imports, and finally determine which external PyPI packages you need to install in order to satisfy dependencies.

pipsalabim.api.report.ask_multiple_pypi(datadict)[source]

Ask the user about which PyPI package will use to satisfy an import.

Parameters:datadict – a dictionary containing modules as keys and a list as values.
Returns:an updated datadict with the answered information from user.

New in version 0.1.0.

pipsalabim.api.report.get_imports(pkgdata)[source]

List modules imported inside of packages provided in pkgdata.

Parameters:pkgdata – a list of tuples containing the name of a package and the directory where its located.
Returns:a list of the modules imported according to the list of packages provided in pkgdata.

New in version 0.1.0.

pipsalabim.api.report.get_messages(datadict)[source]

Generate messages for each type of module in datadict.

Parameters:datadict – a dictionary containing modules as keys and a list as values.
Returns:a dictionary containing messages for each type of module.

New in version 0.1.0.

pipsalabim.api.report.get_module_datadict(basedir)[source]

Process the current directory to get data from packages and modules.

Parameters:basedir – a string containing a path to the directory to be analized.
Returns:a dictionary containing information for each imported module. Like:
{
    'module_a': ['LOCAL'],
    'module_b': ['pypi_package_1'],
    'module_c': ['pypi_package_1', 'pypi_package_2'],
    'module_d': [],
    'module_e': ['STDLIB2.6', 'STDLIB2.7', 'STDLIB3.5'],
    'module_f': ['STDLIB2.7'],
}

New in version 0.1.0.

pipsalabim.api.report.get_modules(pkgdata)[source]

List modules inside packages provided in pkgdata.

Parameters:pkgdata – a list of tuples containing the name of a package and the directory where its located.
Returns:a list of the modules according to the list of packages provided in pkgdata.

New in version 0.1.0.

pipsalabim.api.report.get_package_dirs(path)[source]

List directories containing python packages on path.

Parameters:path – a path pointing to a directory containing python code.
Returns:a list containing directories of packages.

New in version 0.1.0.

pipsalabim.api.report.get_packages(path)[source]

List packages living in path with its directory.

Parameters:path – a path pointing to a directory containing python code.
Returns:a list of tuples containing the name of the package and the package directory. For example:
[
   ('package_a', '/path/to/package_a'),
   ('package_b.module_b', '/path/to/package_b/module_b'),
   ('package_c.module_c', '/path/to/package_c/module_c')
]

New in version 0.1.0.

pipsalabim.api.report.main(**kwargs)[source]

Generate a report to inform about PyPI dependencies.

This command will search your code for unsatisfied dependencies by looking at your import statements. If an import is not satisfied by internal modules or the standard library, then it will query the PyPI module index provided by PyPIContents.

Sometimes, more than one PyPI package will provide the missing module and in such cases, you will be asked to select one from a list of options.

If Pip Sala Bim fails to find a package providing the module you need, it will report it back to you.

Returns:an exit status.

New in version 0.1.0.

pipsalabim.api.update module

pipsalabim.api.update is a module implementing the update command.

Pip Sala Bim needs to keep a local copy of the PyPIContents database to function properly. This command takes care of maintaining it up-to-date.

Please note that this command must be executed manually.

pipsalabim.api.update.download_json(datafile, dataurl)[source]

Download a json file from dataurl, store to datafile and report.

Parameters:
  • datafile – a string containing a path to store the contents of the file downloaded from dataurl.
  • dataurl – a string containing a url to a file.
Returns:

True if operations went OK, False if not.

New in version 0.1.0.

pipsalabim.api.update.main(**kwargs)[source]

Update databases from PyPIContents.

This function downloads the standard library modules index (stdlib.json) and the PyPI modules index (pypi.json) from the PyPIContents project to keep a local copy of these files, which are critical to PiP Sala Bim.

This command makes no comparing or checking prior to download.

Returns:an exit status.

New in version 0.1.0.

pipsalabim.core package

pipsalabim.core contains the core functions of the application.

This package contains functions for managing the download of files, exploring and obtaining imported modules, logging and other minor file operations.

pipsalabim.core.imports module

pipsalabim.core.imports studies the code searching for imports.

This module has advanced programming to inspect and locate imported modules in python source code. It uses python’s own Abstract Syntax Trees (AST) to do such operations.

class pipsalabim.core.imports.ImportVisitor[source]

AST visitor for grabbing the import statements.

ImportVisitor is a node visitor class that walks the abstract syntax tree and calls a visitor function for every import statement found.

Per default the visitor functions for the nodes are 'visit_' + class name of the import node (lowercase). This visitor only has methods for processing Import and ImportFrom types of nodes. When any other type of node is entered, the generic_visit visitor is used instead.

This visitor produces a list of tuples like:

[
    ('MODULE', 'LEVEL'),
    ('MODULE', 'LEVEL'),
    ('MODULE', 'LEVEL'),
    ('MODULE', 'LEVEL'),
    ('MODULE', 'LEVEL')
]
  • MODULE: a string containing the name of the imported module

    (does not include the package to which it belongs).

  • LEVEL is the relative level of the imported module.

    • 0 is the level of a simple import like:

      import MODULE
      from MODULE import NAME
      
    • 1 is the level of a relative import in the same package, for example:

      import .MODULE
      from .MODULE import NAME
      from . import NAME
      
    • 2 is the level of a relative import in the parent package, for example:

      import ..MODULE
      from ..MODULE import NAME
      from .. import NAME
      
__init__()[source]

Initialize this ImportVisitor.

Sets initial empty values for self.modules which will help storing modules.

Returns:an ImportVisitor instance.

New in version 0.1.0.

modules = None

Attribute modules (list): Stores modules as they are found by the visitor methods.

visit(node)[source]

Visit a node.

Parameters:node – an ast object representing a python statement.
Returns:a reference to visit_import if the node type is Import, to visit_importfrom if the node type is ImportFrom or generic_visit for any other node.

New in version 0.1.0.

visit_import(node)[source]

Append node names to self.modules.

Parameters:node – an ast object representing a python statement.

New in version 0.1.0.

visit_importfrom(node)[source]

Append node names and levels to self.modules.

Parameters:node – an ast object representing a python statement.

New in version 0.1.0.

pipsalabim.core.imports.find_imports(package, filename)[source]

Get a list of modules extracted from import statements.

Parameters:
  • package – a string containing a python package to which filename belongs.
  • filename – a string containing a path to a python source code file.
Returns:

a list of modules in absolute form.

New in version 0.1.0.

pipsalabim.core.imports.parse_python_source(filename)[source]

Parse the file filename and convert it to AST.

Parameters:filename – a string containing a path to a python source code file.
Returns:an AST object or None if the file has a syntax error.

New in version 0.1.0.

pipsalabim.core.logger module

pipsalabim.core.logger is the global application logging module.

All modules use the same global logging object. No messages will be emitted until the logger is started.

class pipsalabim.core.logger.ControlableLogger(name=None)[source]

This class represents a logger object that can be started and stopped.

It has a start method which allows you to specify a logging level. The stop method halts output.

__init__(name=None)[source]

Initialize this ControlableLogger.

The name defaults to the application name. Loggers with the same name refer to the same underlying object. Names are hierarchical, e.g. ‘parent.child’ defines a logger that is a descendant of ‘parent’.

Parameters:name – a string containig the logger name.
Returns:a ControlableLogger instance.

New in version 0.1.0.

disabled = None

Attribute disabled (boolean): Stores the current status of the logger.

formatstring = None

Attribute formatstring (string): Stores the string that will be used to format the logger output.

loglevel(level='WARNING')[source]

Set the log level for this logger.

Messages less than the given priority level will be ignored. The default level is ‘WARNING’, which conforms to the *nix convention that a successful run should produce no diagnostic output. Available levels and their suggested meanings:

  • NOTSET: all messages are processed.
  • DEBUG: output useful for developers.
  • INFO: trace normal program flow, especially external
    interactions.
  • WARNING: an abnormal condition was detected that might need
    attention.
  • ERROR: an error was detected but execution continued.
  • CRITICAL: an error was detected and execution was halted.
Parameters:level – a string containing the desired logging level.

New in version 0.1.0.

start()[source]

Start logging with this logger.

Until the logger is started, no messages will be emitted. This applies to all loggers with the same name and any child loggers.

New in version 0.1.0.

stop()[source]

Stop logging with this logger.

Remove available handlers and set active attribute to False.

New in version 0.1.0.

pipsalabim.core.utils module

pipsalabim.core.utils is a utility module.

This module contains several utilities to process information coming from the other modules.

pipsalabim.core.utils.chunk_read(response, chunk_size=8192, report_hook=None)[source]

Download a file by chunks.

Parameters:
  • response – a file object as returned by urlopen.
  • chunk_size – an integer representing the size of the chunks to be downloaded at a time.
  • report_hook – a function to report the progress of the download.
Returns:

a blob containing the downloaded file.

New in version 0.1.0.

pipsalabim.core.utils.chunk_report(downloaded, total)[source]

Print the progress of a download.

Parameters:
  • downloaded – an integer representing the size (in bytes) of data downloaded so far.
  • total – an integer representing the total size (in bytes) of data that needs to be downloaded.

New in version 0.1.0.

pipsalabim.core.utils.custom_sys_path(*args, **kwds)[source]

Context manager to momentarily change sys.path.

Parameters:new_sys_path – a list of paths to overwrite sys.path.

New in version 0.1.0.

pipsalabim.core.utils.fill_with_local(datadict, modules)[source]

Fill datadict if module is found in modules.

Parameters:
  • datadict – a dictionary containing modules as keys and a list as values.
  • modules – a list of modules present in the local python source code.
Returns:

a dictionary containing information about the location of each imported module.

New in version 0.1.0.

pipsalabim.core.utils.fill_with_pypi(datadict, pypidata)[source]

Fill datadict with modules from pypidata if found.

Parameters:
  • datadict – a dictionary containing modules as keys and a list as values.
  • pypidata – a dictionary with the PyPIContents database.
Returns:

an updated dictionary containing information about the location of each imported module.

New in version 0.1.0.

pipsalabim.core.utils.fill_with_stdlib(datadict, stdlibdata)[source]

Fill datadict with modules from stdlibdata if found.

Parameters:
  • datadict – a dictionary containing modules as keys and a list as values.
  • stdlibdata – a dictionary containing the modules of the standard library of each python version.
Returns:

a dictionary containing information about the location of each imported module.

New in version 0.1.0.

pipsalabim.core.utils.find_files(path=None, pattern='*')[source]

Locate all the files matching the supplied pattern in path.

Locate all the files matching the supplied filename pattern in and below the supplied root directory. If no pattern is supplied, all files will be returned.

Parameters:
  • path – a string containing a path where the files will be looked for.
  • pattern – a string containing a regular expression.
Returns:

a list of files matching the pattern within path (recursive).

New in version 0.1.

pipsalabim.core.utils.is_valid_path(path)[source]

Test if path is a valid python path.

Parameters:path – a string containing a path.
Returns:True if path is a valid python path. False otherwise.

New in version 0.1.0.

pipsalabim.core.utils.list_files(path=None, pattern='*')[source]

List files on path (non-recursively).

Locate all the files matching the supplied filename pattern in the first level of the supplied path. If no pattern is supplied, all files will be returned.

Parameters:
  • path – a string containing a path where the files will be looked for.
  • pattern – a string containing a regular expression.
Returns:

a list of files matching the pattern within the first level of path (non-recursive).

New in version 0.1.0.

pipsalabim.core.utils.remove_sys_modules(*args, **kwds)[source]

Context manager to momentarily remove modules from sys.modules.

Parameters:remove – a list of modules to remove from sys.modules.

New in version 0.1.0.

pipsalabim.core.utils.s(s_string)[source]

Convert a byte stream to string working on both python 2 and 3.

Parameters:s_string – a byte stream to convert to string.

New in version 0.1.5.

pipsalabim.core.utils.u(u_string)[source]

Convert a string to unicode working on both python 2 and 3.

Parameters:u_string – a string to convert to unicode.

New in version 0.1.5.