How to write a custom scan handling class for specplot#

Sometimes, it will be obvious that a certain scan macro never generates any plot images, or that the default handling creates a plot that is a poor representation of the data, such as the hklscan where only one of the the axes hkl is scanned. To pick the scanned axis for plotting, it is necessary to prepare custom handling and replace the default handling.

Overview#

It is possible to add in additional handling by writing a Python module. This module creates a subclass of the standard handling, such as LinePlotter, MeshPlotter, or their superclass ImageMaker. The support is added to the macro selection class Selector with code such as in the brief example described below: Change the plot title text in ascan macros:

selector = spec2nexus.specplot.Selector()
selector.add('ascan', Custom_Ascan)
spec2nexus.specplot_gallery.main()

Data Model#

The data to be plotted is kept in an appropriate subclass of PlotDataStructure in attributes show in the next table. The data model is an adaptation of the NeXus NXdata base class. 1

attribute

description

self.signal

name of the dependent data (y axis or image) to be plotted

self.axes

list of names of the independent axes 2

self.data

dictionary with the data, indexed by name (\(Q\) & \(R\))
1

NeXus NXdata base class: https://download.nexusformat.org/doc/html/classes/base_classes/NXdata.html

2

The number of names provided in self.axes is equal to the rank of the signal data (self.data[self.signal]). For 1-D data, self.axes has one name and the signal data is one-dimensional. For 2-D data, self.axes has two names and the signal data is two-dimensional.

Steps#

In all cases, custom handling of a specific SPEC macro name is provided by creating a subclass of ImageMaker and defining one or more of its methods. In the simplest case, certain settings may be changed by calling spec2nexus.specplot.ImageMaker.configure() with the custom values. Examples of further customization are provided below, such as when the data to be plotted is stored outside of the SPEC data file. This is common for images from area detectors.

It may also be necessary to create a subclass of PlotDataStructure to gather the data to be plotted or override the default spec2nexus.specplot.ImageMaker.plottable() method. An example of this is shown with the MeshPlotter and associated MeshStructure classes.

Examples#

A few examples of custom macro handling are provided, some simple, some complex. In each example, decisions have been made about where to provide the desired features.

Change the plot title text in ascan macros#

The SPEC ascan macro is a workhorse and records the scan of a positioner and the measurement of data in a counter. Since this macro name ends with “scan”, the default selection in specplot images this data using the LinePlotter class. Here is a plot of the default handling of data from the ascan macro:

_images/ascan.png

Standard plot of data from ascan macro#

We will show how to change the plot title as a means to illustrate how to customize the handling for a scan macro.

We write Custom_Ascan which is a subclass of LinePlotter. The get_plot_data method is written (overrides the default method) to gain access to the place where we can introduce the change. The change is made by the call to the configure method (defined in the superclass). Here’s the code:

ascan.py example

 1#!/usr/bin/env python
 2
 3'''
 4Plot all scans that used the SPEC `ascan` macro, showing only the scan number (not full scan command)
 5
 6This is a simple example of how to customize the scan macro handling.
 7There are many more ways to add complexity.
 8'''
 9
10import spec2nexus.specplot
11import spec2nexus.specplot_gallery
12
13
14class Custom_Ascan(spec2nexus.specplot.LinePlotter):
15    '''simple customization'''
16    
17    def retrieve_plot_data(self):
18        '''substitute with the data&time the plot was created'''
19        import datetime
20        spec2nexus.specplot.LinePlotter.retrieve_plot_data(self)
21        self.set_plot_subtitle(str(datetime.datetime.now()))
22
23
24def main():
25    selector = spec2nexus.specplot.Selector()
26    selector.add('ascan', Custom_Ascan)
27    spec2nexus.specplot_gallery.main()
28
29
30if __name__ == '__main__':
31    main()
32
33# -----------------------------------------------------------------------------
34# :author:    Pete R. Jemian
35# :email:     prjemian@gmail.com
36# :copyright: (c) 2014-2022, Pete R. Jemian
37#
38# Distributed under the terms of the Creative Commons Attribution 4.0 International Public License.
39#
40# The full license is in the file LICENSE.txt, distributed with this software.
41# -----------------------------------------------------------------------------

See the changed title:

_images/ascan_custom.png

Customized plot of data from ascan macro#

Make the y-axis log scale#

A very simple customization can make the Y axis to be logarithmic scale. (This customization is planned for an added feature 3 in a future relase of the spec2nexus package.) We present two examples.

modify handling of a2scan#

One user wants all the a2scan images to be plotted with a logarithmic scale on the Y axis. Here’s the code:

custom_a2scan_gallery.py example

 1#!/usr/bin/env python
 2
 3'''
 4Customization for specplot_gallery: plot a2scan with log(y) axis
 5
 6This program changes the plotting for all scans that used the *a2scan* SPEC macro.
 7The Y axis of these plots will be plotted as logarithmic if all the data values are 
 8greater than zero.  Otherwise, the Y axis scale will be linear.
 9'''
10
11import spec2nexus.specplot
12import spec2nexus.specplot_gallery
13
14class Custom_a2scan_Plotter(spec2nexus.specplot.LinePlotter):
15    '''plot `a2scan` y axis as log if possible'''
16    
17    def retrieve_plot_data(self):
18        '''plot the vertical axis on log scale'''
19        spec2nexus.specplot.LinePlotter.retrieve_plot_data(self)
20
21        choose_log_scale = False
22
23        if self.signal in self.data:    # log(y) if all data positive
24            choose_log_scale = min(self.data[self.signal]) > 0
25
26        self.set_y_log(choose_log_scale)
27
28
29def main():
30    selector = spec2nexus.specplot.Selector()
31    selector.add('a2scan', Custom_a2scan_Plotter)
32    spec2nexus.specplot_gallery.main()
33
34
35if __name__ == '__main__':
36    # debugging_setup()
37    main()
38
39'''
40Instructions:
41
42Save this file in a directory you can write and call it from your cron tasks.  
43
44Note that in cron entries, you cannot rely on shell environment variables to 
45be defined.  Best to spell things out completely.  For example, if your $HOME 
46directory is `/home/user` and you have these directories:
47
48* `/home/user/bin`: various custom executables you use
49* `/home/user/www/specplots`: a directory you access with a web browser for your plots
50* `/home/user/spec/data`: a directory with your SPEC data files
51
52then save this file to `/home/user/bin/custom_a2scan_gallery.py` and make it executable
53(using `chmod +x ./home/user/bin/custom_a2scan_gallery.py`).
54
55Edit your list of cron tasks using `crontab -e` and add this (possibly 
56replacing a call to `specplot_gallery` with this call `custom_a2scan_gallery.py`)::
57
58    # every five minutes (generates no output from outer script)
59    0-59/5 * * * *  /home/user/bin/custom_a2scan_gallery.py -d /home/user/www/specplots /home/user/spec/data 2>&1 >> /home/user/www/specplots/log_cron.txt
60
61Any output from this periodic task will be recorded in the file
62`/home/user/www/specplots/log_cron.txt`.  This file can be reviewed
63for diagnostics or troubleshooting.
64'''

custom uascan#

The APS USAXS instrument uses a custom scan macro called uascan for routine step scans. Since this macro name ends with “scan”, the default selection in specplot images this data using the LinePlotter class. Here is a plot of the default handling of data from the uascan macro:

_images/uascan_as_ascan.png

USAXS uascan, handled as LinePlotter#

The can be changed by making the y axis log scale. To do this, a custom version of LinePlotter is created as Custom_Ascan. The get_plot_data method is written (overrides the default method) to make the y axis log-scale by calling the configure method (defined in the superclass). Here’s the code:

usaxs_uascan.py example

 1#!/usr/bin/env python
 2
 3"""
 4Plot data from the USAXS uascan macro
 5
 6.. autosummary::
 7
 8    ~UAscan_Plotter
 9
10"""
11
12import spec2nexus.specplot
13import spec2nexus.specplot_gallery
14
15
16class UAscan_Plotter(spec2nexus.specplot.LinePlotter):
17    """Customized `uascan` handling"""
18
19    def retrieve_plot_data(self):
20        """plot the vertical axis on log scale"""
21        spec2nexus.specplot.LinePlotter.retrieve_plot_data(self)
22
23        if self.signal in self.data:
24            # can't plot negative Y on log scale
25            # Alternative to raising NotPlottable would be
26            # to remove any data where Y <= 0
27            if min(self.data[self.signal]) <= 0:
28                msg = "cannot plot Y<0: " + str(self.scan)
29                raise spec2nexus.specplot.NotPlottable(msg)
30
31        # in the uascan, a name for the sample is given in `self.scan.comments[0]`
32        self.set_y_log(True)
33        self.set_plot_subtitle(
34            "#%s uascan: %s" % (str(self.scan.scanNum), self.scan.comments[0])
35        )
36
37
38def debugging_setup():
39    import os, sys
40    import shutil
41    import ascan
42
43    selector = spec2nexus.specplot.Selector()
44    selector.add("ascan", ascan.Custom_Ascan)  # just for the demo
45    path = "__usaxs__"
46    shutil.rmtree(path, ignore_errors=True)
47    os.mkdir(path)
48    sys.argv.append("-d")
49    sys.argv.append(path)
50    sys.argv.append(
51        os.path.join("..", "src", "spec2nexus", "data", "APS_spec_data.dat")
52    )
53
54
55def main():
56    selector = spec2nexus.specplot.Selector()
57    selector.add("uascan", UAscan_Plotter)
58    spec2nexus.specplot_gallery.main()
59
60
61if __name__ == "__main__":
62    # debugging_setup()
63    main()
64
65# -----------------------------------------------------------------------------
66# :author:    Pete R. Jemian
67# :email:     prjemian@gmail.com
68# :copyright: (c) 2014-2022, Pete R. Jemian
69#
70# Distributed under the terms of the Creative Commons Attribution 4.0 International Public License.
71#
72# The full license is in the file LICENSE.txt, distributed with this software.
73# -----------------------------------------------------------------------------

Note that in the uascan, a name for the sample provided by the user is given in self.scan.comments[0]. The plot title is changed to include this and the scan number. The customized plot has a logarithmic y axis:

_images/uascan_log_y.png

USAXS uascan, with logarithmic y axis#

The most informative view of this data is when the raw data are reduced to \(I(Q)\) and viewed on a log-log plot, but that process is beyond this simple example. See the example Get xy data from HDF5 file below.

3

specplot: add option for default log(signal)

SPEC’s hklscan macro#

The SPEC hklscan macro appears in a SPEC data file due to either a hscan, kscan, or lscan. In each of these one of the hkl vectors is scanned while the other two remain constant.

The normal handling of the ascan macro plots the last data column against the first. This works for data collected with the hscan. For kscan or lscan macros, the h axis is still plotted by default since it is in the first column.

_images/hklscan_as_ascan.png

SPEC hklscan (lscan, in this case), plotted against the (default) first axis H#

To display the scanned axis, it is necessary to examine the data in a custom subclass of LinePlotter. The HKLScanPlotter subclass, provided with specplot, defines the get_plot_data() method determines the scanned axis, setting it by name:

plot.axes = [axis,]
self.scan.column_first = axis

Then, the standard plot handling used by LinePlotter uses this information to make the plot.

_images/hklscan.png

SPEC hklscan (lscan), plotted against L#

Get xy data from HDF5 file#

One example of complexity is when SPEC has been used to direct data collection but the data is not stored in the SPEC data file. The SPEC data file scan must provide some indication about where the collected scan data has been stored.

custom usaxs_flyscan#

The USAXS instrument at APS has a FlyScan macro that commands the instrument to collect data continuously over the desired \(Q\) range. The data is written to a NeXus HDF5 data file. Later, a data reduction process converts the arrays of raw data to one-dimensional \(I(Q)\) profiles. The best representation of this reduced data is on a log-log plot to reveal the many decades of both \(I\) and \(Q\) covered by the measurement.

With the default handling by LinePlotter, no plot can be generated since the data is given in a separate HDF5 file. That file is read with the custom handling of the usaxs_flyscan.py demo:

usaxs_flyscan.py example

  1#!/usr/bin/env python
  2
  3"""
  4Plot data from the USAXS FlyScan macro.
  5
  6.. autosummary::
  7
  8    ~read_reduced_fly_scan_file
  9    ~retrieve_flyScanData
 10    ~USAXS_FlyScan_Structure
 11    ~USAXS_FlyScan_Plotter
 12
 13"""
 14
 15import h5py
 16import numpy
 17import pathlib
 18
 19import spec2nexus.specplot
 20import spec2nexus.specplot_gallery
 21
 22# $URL: https://subversion.xray.aps.anl.gov/small_angle/USAXS/livedata/specplot.py $
 23REDUCED_FLY_SCAN_BINS = 250  # the default
 24PLOT_AXES = ["Q", "R"]
 25
 26
 27# methods picked (& modified) from the USAXS livedata project
 28def read_reduced_fly_scan_file(hdf5_file_name):
 29    """
 30    Read any and all reduced data from the HDF5 file, return in a dictionary.
 31
 32    dictionary = {
 33      'full': (dictionary keys: Q, R, R_max, ar, fwhm, centroid)
 34      '250':  (dictionary keys: Q, R, dR)
 35      '5000': (dictionary keys: Q, R, dR)
 36    }
 37    """
 38
 39    reduced = {}
 40    with h5py.File(str(hdf5_file_name), "r") as hdf:
 41        entry = hdf["/entry"]
 42        for key in entry.keys():
 43            if key.startswith("flyScan_reduced_"):
 44                nxdata = entry[key]
 45                d = {}
 46                for dsname in PLOT_AXES:
 47                    if dsname in nxdata:
 48                        value = nxdata[dsname]
 49                        if value.size == 1:
 50                            d[dsname] = float(value[0])
 51                        else:
 52                            d[dsname] = numpy.array(value)
 53                reduced[key[len("flyScan_reduced_") :]] = d
 54    return reduced
 55
 56
 57def retrieve_flyScanData(scan):
 58    """Retrieve reduced, rebinned data from USAXS Fly Scans."""
 59    comment = scan.comments[2]
 60    key_string = "FlyScan file name = "
 61    index = comment.find(key_string) + len(key_string)
 62
 63    hdf_file_name = comment[index:-1]
 64    path = pathlib.Path(scan.header.parent.fileName).parent
 65    abs_file = (path / hdf_file_name).absolute()
 66
 67    plotData = {}
 68    if abs_file.exists():
 69        reduced = read_reduced_fly_scan_file(abs_file)
 70        s_num_bins = str(REDUCED_FLY_SCAN_BINS)
 71
 72        choice = reduced.get(s_num_bins) or reduced.get("full")
 73
 74        if choice is not None:
 75            plotData = {axis: choice[axis] for axis in PLOT_AXES}
 76
 77    return plotData
 78
 79
 80class USAXS_FlyScan_Plotter(spec2nexus.specplot.LinePlotter):
 81    """
 82    Customize `FlyScan` handling, plot :math:`log(I)` *vs.* :math:`log(Q)`.
 83
 84    The USAXS FlyScan data is stored in a NeXus HDF5 file in a subdirectory
 85    below the SPEC data file.  This code uses existing code from the
 86    USAXS instrument to read that file.
 87    """
 88
 89    def retrieve_plot_data(self):
 90        """Retrieve reduced data from the FlyScan's HDF5 file."""
 91        # get the data from the HDF5 file
 92        fly_data = retrieve_flyScanData(self.scan)
 93
 94        if len(fly_data) != 2:
 95            raise spec2nexus.specplot.NoDataToPlot(str(self.scan))
 96
 97        self.signal = "R"
 98        self.axes = ["Q"]
 99        self.data = fly_data
100
101        # customize the plot just a bit
102        # sample name as given by the user?
103        subtitle = "#" + str(self.scan.scanNum)
104        subtitle += " FlyScan: " + self.scan.comments[0]
105        self.set_plot_subtitle(subtitle)
106        self.set_x_log(True)
107        self.set_y_log(True)
108        self.set_x_title(r"$|\vec{Q}|, 1/\AA$")
109        self.set_y_title(r"USAXS $R(|\vec{Q}|)$, a.u.")
110
111    def plottable(self):
112        """
113        Can this data be plotted as expected?
114        """
115        if self.signal in self.data:
116            signal = self.data[self.signal]
117            if signal is not None and len(signal) > 0 and len(self.axes) == 1:
118                if len(signal) == len(self.data[self.axes[0]]):
119                    return True
120        return False
121
122
123def debugging_setup():
124    import sys
125    import shutil
126
127    path = pathlib.Path("..") / "src"
128    sys.path.insert(0, str(path))
129    path = "__usaxs__"
130    shutil.rmtree(path, ignore_errors=True)
131    pathlib.os.mkdir(path)
132    sys.argv.append("-d")
133    sys.argv.append(path)
134    data_file = path / "spec2nexus" / "data" / "02_03_setup.dat"
135    sys.argv.append(str(data_file.absolute()))
136
137
138def main():
139    selector = spec2nexus.specplot.Selector()
140    selector.add("FlyScan", USAXS_FlyScan_Plotter)
141    spec2nexus.specplot_gallery.main()
142
143
144if __name__ == "__main__":
145    # debugging_setup()
146    main()
147
148# -----------------------------------------------------------------------------
149# :author:    Pete R. Jemian
150# :email:     prjemian@gmail.com
151# :copyright: (c) 2014-2022, Pete R. Jemian
152#
153# Distributed under the terms of the Creative Commons Attribution 4.0 International Public License.
154#
155# The full license is in the file LICENSE.txt, distributed with this software.
156# -----------------------------------------------------------------------------

The data is then rendered in a customized log-log plot of \(I(Q)\):

_images/usaxs_flyscan.png

USAXS FlyScan, handled by USAXS_FlyScan_Plotter#

The USAXS_FlyScan_Plotter() class provides custom methods for retrieve_plot_data() and plottable() which will be called from spec2nexus.specplot.ImageMaker.plot_scan().

Method USAXS_FlyScan_Plotter.plottable() returns a boolean value if the data can be plotted.

Method USAXS_FlyScan_Plotter.retrieve_plot_data() gets the data from the scan by calling retrieve_flyScanData() with the scan object. Then the method customizes the plot details.

Function retrieve_flyScanData(scan) gets the name of the NeXus/HDF5 file from the scan and reads the HDF5 file, returning either the reduced data with the number of points (as described in REDUCED_FLY_SCAN_BINS) or the full data set.

Usage#

When a custom scan macro handler is written and installed using code similar to the custom ascan handling above:

def main():
    selector = spec2nexus.specplot.Selector()
    selector.add('ascan', Custom_Ascan)
    spec2nexus.specplot_gallery.main()


if __name__ == '__main__':
    main()

then the command line arugment handling from spec2nexus.specplot_gallery.main() can be accessed from the command line for help and usage information.

Usage:

user@localhost ~/.../spec2nexus/demo $ ./ascan.py
usage: ascan.py [-h] [-r] [-d DIR] paths [paths ...]
ascan.py: error: too few arguments

Help:

user@localhost ~/.../spec2nexus/demo $ ./ascan.py -h
usage: ascan.py [-h] [-r] [-d DIR] paths [paths ...]

read a list of SPEC data files (or directories) and plot images of all scans

positional arguments:
  paths              SPEC data file name(s) or directory(s) with SPEC data
                     files

optional arguments:
  -h, --help         show this help message and exit
  -r                 sort images from each data file in reverse chronolgical
                     order
  -d DIR, --dir DIR  base directory for output (default:/home/prjemian/Documen
                     ts/eclipse/spec2nexus/demo)