Jupyter Tidbit: IPython's ! returns an SList
September 15, 2018
This post originates from a gist that supports comments, forks, and execution in binder.
Summary
IPython shell assignment (the ! operator) evaluates a command using the local shell (e.g., bash) and returns a string list (IPython.utils.text.SList). An SList is a list-like object containing "chunks" of stdout and stderr, properties for accessing those chunks in different forms, and convenience methods for operating on them.
Example
The SList.ipynb notebook below uses SList properties to access the output of a shell command as a list-like of strings, a newline-separated string, a space-separated string, and a list of pathlib.Path objects. The notebook then uses the SList.fields() and SList.grep() methods to extract columns from and search command output.
Why is this useful?
You can take advantage of the properties and methods of an SList to transform shell output into forms more amenable to further operations in Python.
For more information
See the IPython documentation about Shell Assignment for examples of executing shell commands with optional Python values as inputs. See the IPython documentation about String Lists for additional demontrations of the utility of SLists.
SList.ipynb
Start with a simple ls command.
ls = !ls
Note that the return type is not a simple list.
type(ls)
It is an SList, a list-like object that contains "chunks" of stdout and stderr, properties for accessing those chunks in different forms, and convenience methods for operating on them.
ls
There are many ways to refer to the output from an SList as a list-like of strings.
ls == ls.get_list() == ls.list == ls.l
Some properties also return the output as a newline delimited string.
print(ls.nlstr)
ls.get_nlstr() == ls.nlstr == ls.n
Other properties return the output as a space separated string.
print(ls.spstr)
ls.get_spstr() == ls.spstr == ls.s
Still other properties return the output as a list of pathlib.Path instances.
ls.paths
ls.get_paths() == ls.paths == ls.p
import pathlib
isinstance(ls.paths[0], pathlib.Path)
These are convenient for performing further path operations in Python.
[p.is_dir() for p in ls.paths]
SList objects expose a fields() method.
df = !df -h
df
fields splits the output into whitespace delimited columns and returns the values of columns, specified by their indices, as space-separated strings.
df.fields(0,4)
SList objects also expose a grep() method. grep evaluates a regular expression or callable against all elements of the SList or a whitespace delimited column in each element.
df.grep('dev', field=5)
grep(prune=True) turns the grep into a filtering operation instead of a matching operation.
hosts = !cat /etc/hosts
print(hosts.n)
The return value of grep (and fields) is another SList supporting all of the features noted above.
print(hosts.grep('ip6', prune=True).n)