spec2nexus.spec#

Library of classes to read the contents of a SPEC data file.

How to use spec2nexus.spec#

spec2nexus.spec provides Python support to read the scans in a SPEC data file. (It does not provide a command-line interface.) Here is a quick example how to use spec:

1from spec2nexus.spec import SpecDataFile
2
3specfile = SpecDataFile('data/33id_spec.dat')
4print 'SPEC file name:', specfile.specFile
5print 'SPEC file time:', specfile.headers[0].date
6print 'number of scans:', len(specfile.scans)
7
8for scanNum, scan in specfile.scans.items():
9    print scanNum, scan.scanCmd

For one example data file provided with spec2nexus.spec, the output starts with:

1SPEC file name: samplecheck_7_17_03
2SPEC file time: Thu Jul 17 02:37:32 2003
3number of scans: 106
41  ascan  eta 43.6355 44.0355  40 1
52  ascan  chi 73.47 73.87  40 1
63  ascan  del 84.6165 84.8165  20 1
74  ascan  del 84.5199 84.7199  20 1
85  ascan  del 84.3269 84.9269  30 1
9...

How to read one scan#

Here is an example how to read one scan:

1from spec2nexus.spec import SpecDataFile
2
3specfile = SpecDataFile('data/33id_spec.dat')
4specscan = specfile.getScan(5)
5print specscan.scanNum
6print specscan.scanCmd

which has this output:

5
ascan  del 84.3269 84.9269  30 1

Alternatively, it is possible to use choose a scan using [scan_number] syntax. This code is equivalent to the above:

from spec2nexus.spec import SpecDataFile

specfile = SpecDataFile('data/33id_spec.dat')
specscan = specfile[5]
print specscan.scanNum
print specscan.scanCmd

The data columns are provided in a dictionary. Using the example above, the dictionary is specscan.data where the keys are the column labels (from the #L line) and the values are from each row. It is possible to make a default plot of the last column vs. the first column. Here’s how to find that data:

1x_label = specscan.L[0]          # first column from #L line
2y_label = specscan.L[-1]         # last column from #L line
3x_data = specscan.data[x_label]  # data for first column
4y_data = specscan.data[y_label]  # data for last column

Get a list of the scans#

The complete list of scan numbers from the data file is obtained, sorted alphabetically by scan number:

all_scans = specfile.getScanNumbers()

Same list sorted by date & time:

all_scans = specfile.getScanNumbersChronological()

Select from the list of the scans#

Get a scan (or list of scans) by slicing from the SpecDataFile() object.

EXAMPLES:

First, read a SPEC data file:

>>> from spec2nexus.spec import SpecDataFile
>>> specfile = SpecDataFile('data/CdOsO')

Show scan number 5:

>>> specscan = specfile[5]
>>> print(specscan)
5  ascan  stblx 1.925 3.925  50 1

Show the last scan:

>>> print(specfile[-1])
73  ascan  micro_y 6.7515 7.1515  20 1

Show scan numbers below 4. Since the slice uses : characters, it returns a list of scans:

>>> for specscan in specfile[:4]:
...    print(specscan)
1  ascan  herixE -17.3368 -7.33676  40 1
1  ascan  herixE -1.00571 18.9943  20 30
2  ascan  herixE -22.3505 -12.3505  40 1
3  ascan  micro_y 4.75 6.75  50 1

Note there are two scans with scan number 1. Slice the first scan number 1:

>>> for specscan in specfile[1:2:0]:
...    print(specscan)
1  ascan  herixE -17.3368 -7.33676  40 1

Slice the second scan number 1:

>>> for specscan in specfile[1:2:1]:
...    print(specscan)
1  ascan  herixE -1.00571 18.9943  20 30

Slice the last one scan number (same result):

>>> for specscan in specfile[1:2:-1]:  # slice the last one (same result)
...    print(specscan)
1  ascan  herixE -1.00571 18.9943  20 30

Compare slicing methods. Same scan:

>>> specfile.getScan(1.1) == specfile[1.1]
True

These are different scans:

>>> specfile.getScan(1.1) == specfile[1]
False

Note that getScan() returns a single scan while slicing (using :) returns a list. This fails:

>>> specfile.getScan(1.1) == specfile[1:2:-1]
False

and this succeeds:

>>> [specfile.getScan(1.1)] == specfile[1:2:-1]
True

Slice Parameters#

When slicing the data file object for scans (specfile[given]), consider that the given slice has the parameters in the following table:

slice

meaning

i

one scan referenced by i (or None if not found)

start:

scan list from start (start <= scan_number)

:finish

scan list up to stop (scan_number < stop)

::which

scan list selecting from identical #S scan_number

start:finish:which

scan list with full slicing syntax

empty

raises SyntaxError if []

None

raises TypeError if [None]

start:finish

raises IndexError if start and finish have opposite signs (can’t mix relative and absolute slicing)
  • start & finish take these meanings:

    • None : match all

    • string : will be converted to integer

    • >=0 : match by scan number (from SPEC data file #S lines)

    • <0 : match by relative position in the list of scans from getScanNumbersChronological()

  • which takes these meanings:

    • None : match all

    • string : will be converted to integer

    • <0 : match by relative position in the list of duplicated scan numbers

    • >=0 : match by specific number in the list of duplicated scan numbers

    Example: which=1 will only match scan numbers with .1 while which=-1 will match the last of every scan number.

Many duplicated scan numbers

When there are many duplicates of a scan number, it may be necessary to use a string representation to distinguish between, for example .1 (second occurrence) and .10 (eleventh occurrence).

Examples:

>>> specfile['4.10'] != specfile[4.10]
False
>>> specfile['4.1'] == specfile[4.10]
True

SPEC data files#

The SPEC data file format is described in the SPEC manual. 1 This manual is taken as a suggested starting point for most users. Data files with deviations from this standard are produced at some facilities.

1

SPEC manual: https://www.certif.com/spec_manual/user_1_4_1.html

Assumptions about data file structure#

These assumptions are used to parse SPEC data files:

  1. SPEC data files are text files organized by lines. The lines can be categorized as: control lines, data lines, and blank lines.

    line type

    description

    control

    contain a # character in the first column followed by a command word 2

    data

    generally contain a row of numbers (the scan data)

    special data

    containing MCA data 3

  2. Lines in a SPEC data file start with a file name control line, then series of blocks. Each block may be either a file header block or a scan block. (Most SPEC files have only one header block. A new header block is created if the list of positioners is changed in SPEC without creating a new file. SPEC users are encouraged to always start a new data file after changing the list of positioners.) A block consists of a series of control, data, and blank lines.

    SPEC data files are composed of a sequence of a single file header block and zero or more scan blocks. 4

  3. A SPEC data file always begins with this control lines: #F, such as:

    #F samplecheck_7_17_03

  4. A file header block begins with these control lines in order: #E #D #C, such as:

    #E 1058427452
#D Thu Jul 17 02:37:32 2003
#C psic  User = epix

  5. A scan block begins with these command lines in order: #S #D, such as:

    #S 78  ascan  del 84.6484 84.8484  20 1
#D Thu Jul 17 08:03:54 2003
2

See Example of Control Lines

3

See Example of MCA data lines

4

It is very unusual to have more than one file header block in a SPEC data file.

Control lines (keys) defined by SPEC#

Here is a list 5 of keys (command words) from the comments in the file.mac (SPEC v6) macro source file:

command word

description

#C

comment line

#D date

current date and time in UNIX format

#E num

the UNIX epoch (seconds from 00:00 GMT 1/1/70)

#F name

name by which file was created

#G1 …

geometry parameters from G[] array (geo mode, sector, etc)

#G2 …

geometry parameters from U[] array (lattice constants, orientation reflections)

#G3 …

geometry parameters from UB[] array (orientation matrix)

#G4 …

geometry parameters from Q[] array (lambda, frozen angles, cut points, etc)

#I num

a normalizing factor to apply to the data

#j% …

mnemonics of counter (% = 0,1,2,… with eight counters per row)

#J% …

names of counters (each separated by two spaces)

#L s1 …

labels for the data columns

#M num

data was counted to this many monitor counts

#N num [num2]

number of columns of data [ num2 sets per row ]

#o% …

mnemonics of motors (% = 0,1,2,… with eight motors per row)

#O% …

names of motors (each separated by two spaces)

#P% …

positions of motors corresponding to above #O/#o

#Q

a reciprocal space position (H K L)

#R

user-defined results from a scan

#S num

scan number

#T num

data was counted for this many seconds

#U

user defined

#X

a temperature

#@MCA fmt

this scan contains MCA data (array_dump() format, as in "%16C")

#@CALIB a b c

coefficients for x[i] = a + b * i + c * i * i for MCA data

#@CHANN n f l r

MCA channel information (number_saved, first_saved, last_saved, reduction coef)

#@CTIME p l r

MCA count times (preset_time, elapsed_live_time, elapsed_real_time)

#@ROI n f l

MCA ROI channel information (ROI_name, first_chan, last_chan)
5

Compare with Supplied spec plugin modules

Example of Control Lines#

The command word of a control line may have a number at the end, indicating it is part of a sequence, such as these control lines (see Control lines (keys) defined by SPEC for how to interpret):

 1#D Wed Nov 03 13:42:03 2010
 2#T 0.3  (seconds)
 3#G0 0
 4#G1 0
 5#G3 0
 6#G4 0
 7#Q
 8#P0 -0.5396381 -0.5675 0.395862 0.7425 40.489861 0 5.894899e-07 11
 9#P1 24 0 -1.19 9.0028278 25.000378 -22.29064 1.5 5
10#P2 -43 -0.01 98 11.8 0 -6.3275 111.52875 -8.67896
11#P3 -0.11352 1e-05 0.199978 0.4001875 1.2998435 15.6077 0 0
12#P4 3.03 0 3.21 6.805 2.835 2.4475 0.9355 -0.072
13#P5 1.31 0.0875 2442.673 -0.391 12 -14.4125 15.498553

Example of MCA data lines#

Lines with MCA array data begin with the @A command word. (If such a data line ends with a continuation character \, the next line is read as part of this line.)

This is an example of a 91-channel MCA data array with trivial (zero) values:

1@A 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
6 0 0 0 0 0 0 0 0 0 0 0

Several MCA spectra may be written to a scan. In this case, a number follows @A indicating which spectrum, such as in this example with four spectra:

1     @A1 0 0 0 0 0 0 35 0 0 35
2     @A2 0 0 0 0 0 0 0 35 0 35
3     @A3 0 0 35 35 0 0 0 0 0 0
4     @A4 0 0 0 0 0 35 35 0 35 0

Supported header keys (command words)#

The SPEC data file keys recognized by spec are listed in Supplied spec plugin modules.

source code summary#

classes#

SpecDataFile

contents of a SPEC data file

SpecDataFileHeader

contents of a spec data file header (#E) section

SpecDataFileScan

contents of a spec data file scan (#S) section

methods#

strip_first_word

return everything after the first space on the line from the spec data file

is_spec_file

test if a given file name is a SPEC data file

exceptions#

SpecDataFileNotFound

data file was not found

SpecDataFileCouldNotOpen

data file could not be opened

SpecDataFileNotFound

data file was not found

DuplicateSpecScanNumber

multiple use of scan number in a single SPEC data file

UnknownSpecFilePart

unknown part in a single SPEC data file

dependencies#

os

OS routines for NT or Posix depending on what system we're on.

re

Support for regular expressions (RE).

sys

This module provides access to some objects used or maintained by the interpreter and to functions that interact strongly with the interpreter.

internal structure of spec2nexus.spec.SpecDataFileScan#

The internal variables of a Python class are called attributes. It may be convenient, for some, to think of them as variables.

scan attributes#

parent

obj - instance of spec2nexus.spec.SpecDataFile

scanNum

int - SPEC scan number

scanCmd

str - SPEC command line

raw

str - text of scan, as reported in SPEC data file

scan attributes (variables) set after call to plugins#

These attributes are only set after the scan’s interpret() method is called. This method is called automatically when trying to read any of the following scan attributes:

comments

[str] - list of all comments reported in this scan

data

{label,[number]} - written by spec2nexus.plugins.spec_common_spec2nexus.data_lines_postprocessing()

data_lines

[str] - raw data (and possibly MCA) lines with comment lines removed

date

str - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Date

G

{key,[number]} - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Geometry

I

float - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_NormalizingFactor

header

obj - instance of spec2nexus.spec.SpecDataFileHeader

L

[str] - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Labels

M

str - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Monitor

positioner

{key,number} - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Positioners.postprocess

N

[int] - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_NumColumns

P

[str] - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Positioners

Q

[number] - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_HKL

S

str - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Scan

T

str - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_CountTime

V

{key,number|str} - written by spec2nexus.plugins.unicat_spec2nexus.UNICAT_MetadataValues

column_first

str - label of first (ordinate) data column

column_last

str - label of last (abscissa) data column

internal use only - do not modify#

These scan attributes are for internal use only and are not part of the public interface. Do not modify them or write code that depends on them.

postprocessors

{key,obj} - dictionary of postprocessing methods

h5writers

{key,obj} - dictionary of methods that write HDF5 structure

__lazy_interpret__

bool - Is lazy (on-demand) call to interpret() needed?

__interpreted__

bool - Has interpret() been called?

source code documentation#

A set of classes to read the contents of a SPEC data file.

author

Pete Jemian

email

jemian@anl.gov

SpecDataFile() is the only class users will need to call. All other spec classes are called from this class. The read() method is called automatically.

The user should create a class instance for each spec data file, specifying the file reference (by path reference as needed) and the internal routines will take care of all that is necessary to read and interpret the information.

is_spec_file(filename)

test if a given file name is a SPEC data file

is_spec_file_with_header(filename)

test if a given file name is a SPEC data file

SpecDataFile(filename)

contents of a SPEC data file

SpecDataFileHeader(buf[, parent])

contents of a spec data file header (#E) section

SpecDataFileScan(header, buf[, parent])

contents of a spec data file scan (#S) section

Note that the SPEC geometry control lines (#G0 #G1 …) have meanings that are unique to specific diffractometer geometries including different numbers of values. Consult the geometry macro file for specifics.

Examples

Get the first and last scan numbers from the file:

>>> from spec2nexus import spec
>>> spec_data = spec.SpecDataFile('path/to/my/spec_data.dat')
>>> print(spec_data.fileName)
path/to/my/spec_data.dat
>>> print('first scan: ', spec_data.getFirstScanNumber())
1
>>> print('last scan: ', spec_data.getLastScanNumber())
22

Get plottable data from scan number 10:

>>> from spec2nexus import spec
>>> spec_data = spec.SpecDataFile('path/to/my/spec_data.dat')
>>> scan10 = spec_data.getScan(10)
>>> x_label = scan10.L[0]
>>> y_label = scan10.L[-1]
>>> x_data = scan10.data[x_label]
>>> y_data = scan10.data[y_label]

Try to read a file that does not exist:

>>> spec_data = spec.SpecDataFile('missing_file')
Traceback (most recent call last):
  ...
spec.SpecDataFileNotFound: file does not exist: missing_file

Classes and Methods

exception spec2nexus.spec.DuplicateSpecScanNumber[source]#

multiple use of scan number in a single SPEC data file

exception spec2nexus.spec.NotASpecDataFile[source]#

content of file is not SPEC data (first line must start with #F)

class spec2nexus.spec.SpecDataFile(filename)[source]#

contents of a SPEC data file

dissect_file()

divide (SPEC data file text) buffer into sections

getFirstScanNumber()

return the first scan

getLastScanNumber()

return the last scan

getMaxScanNumber()

return the highest numbered scan

getMinScanNumber()

return the lowest numbered scan

getScan([scan_number])

return the scan number indicated, None if not found

getScanCommands([scan_list])

return all the scan commands as a list, with scan number

getScanNumbers()

return a list of all scan numbers sorted by scan number

getScanNumbersChronological()

return a list of all scan numbers sorted by date

read()

Reads and parses a spec data file

refresh()

update (refresh) the content if the file is updated

update_available

Has the file been updated since the last time it was read?
dissect_file()[source]#

divide (SPEC data file text) buffer into sections

internal: A section starts with either #F | #E | #S

RETURNS

[block]

list of sections where each section is one or more lines of text with one of the above control lines at its start

getFirstScanNumber()[source]#

return the first scan

getLastScanNumber()[source]#

return the last scan

getMaxScanNumber()[source]#

return the highest numbered scan

getMinScanNumber()[source]#

return the lowest numbered scan

getScan(scan_number=0)[source]#

return the scan number indicated, None if not found

getScanCommands(scan_list=None)[source]#

return all the scan commands as a list, with scan number

getScanNumbers()[source]#

return a list of all scan numbers sorted by scan number

getScanNumbersChronological()[source]#

return a list of all scan numbers sorted by date

read()[source]#

Reads and parses a spec data file

refresh()[source]#

update (refresh) the content if the file is updated

returns previous last_scan or None if file not updated

property update_available#

Has the file been updated since the last time it was read?

Reference file modification time is stored after file is read in read() method.

EXAMPLE USAGE

Open the SPEC data file (example):

sdf = spec.SpecDataFile(filename)

then, monitor (continuing example):

if sdf.update_available:

myLastScan = sdf.last_scan sdf.read() plot_scan_and_newer(myLastScan) # new method myLastScan = sdf.last_scan

exception spec2nexus.spec.SpecDataFileCouldNotOpen[source]#

data file could not be opened

class spec2nexus.spec.SpecDataFileHeader(buf, parent=None)[source]#

contents of a spec data file header (#E) section

interpret()

Interpret the supplied buffer with the spec data file header.

addPostProcessor(label, func)

add a function to be processed after interpreting all lines from a header

addH5writer(label, func)

add a function to be processed when writing the scan header

getLatestScan()

addH5writer(label, func)[source]#

add a function to be processed when writing the scan header

Parameters

  • label (str) – unique label by which this writer will be known

  • func (obj) – function reference of writer

The writers will be called when the HDF5 file is to be written.

addPostProcessor(label, func)[source]#

add a function to be processed after interpreting all lines from a header

Parameters

  • label (str) – unique label by which this postprocessor will be known

  • func (obj) – function reference of postprocessor

The postprocessors will be called at the end of header interpretation.

interpret()[source]#

Interpret the supplied buffer with the spec data file header.

exception spec2nexus.spec.SpecDataFileNotFound[source]#

data file was not found

class spec2nexus.spec.SpecDataFileScan(header, buf, parent=None)[source]#

contents of a spec data file scan (#S) section

get_macro_name()

name of the SPEC macro used for this scan

interpret()

Interpret the supplied buffer with the spec scan data.

add_interpreter_comment(comment)

allow the interpreter to communicate information to the caller

get_interpreter_comments()

return the list of comments

addPostProcessor(label, func)

add a function to be processed after interpreting all lines from a scan

addH5writer(label, func)

add a function to be processed when writing the scan data
addH5writer(label, func)[source]#

add a function to be processed when writing the scan data

Parameters

  • label (str) – unique label by which this writer will be known

  • func (obj) – function reference of writer

The writers will be called when the HDF5 file is to be written.

addPostProcessor(label, func)[source]#

add a function to be processed after interpreting all lines from a scan

Parameters

  • label (str) – unique label by which this postprocessor will be known

  • func (obj) – function reference of postprocessor

The postprocessors will be called at the end of scan data interpretation.

add_interpreter_comment(comment)[source]#

allow the interpreter to communicate information to the caller

see issue #66: https://github.com/prjemian/spec2nexus/issues/66

get_interpreter_comments()[source]#

return the list of comments

see issue #66: https://github.com/prjemian/spec2nexus/issues/66

get_macro_name()[source]#

name of the SPEC macro used for this scan

interpret()[source]#

Interpret the supplied buffer with the spec scan data.

exception spec2nexus.spec.UnknownSpecFilePart[source]#

unknown part in a single SPEC data file

spec2nexus.spec.is_spec_file(filename)[source]#

test if a given file name is a SPEC data file

Parameters

filename (str) – path/to/possible/spec/data.file

filename is a SPEC file if it contains at least one #S control line

spec2nexus.spec.is_spec_file_with_header(filename)[source]#

test if a given file name is a SPEC data file

Parameters

filename (str) – path/to/possible/spec/data.file

filename is a SPEC file only if the file starts 6 with these control lines in order:

  • #F - original filename

  • #E - the UNIX epoch (seconds from 00:00 GMT 1/1/70)

  • #D - current date and time in UNIX format

  • #C - comment line (the first one provides the filename again and the user name)

such as:

#F LNO_LAO
#E 1276730676
#D Wed Jun 16 18:24:36 2010
#C LNO_LAO  User = epix33bm
6

SPEC manual, Standard Data File Format, https://www.certif.com/spec_manual/user_1_4_1.html