Interpret DQ flags

The reftools.interpretdq module contains functions for interpreting DQ flags.

Examples

>>> from reftools.interpretdq import ImageDQ, DQParser

Create a class instance for HST/ACS image using pre-defined DQ definitions. Then, display the associated metadata and translation table:

>>> acsdq = ImageDQ.from_fits('j12345678q_flt.fits', ext=('DQ', 2))  
>>> acsdq.parser.metadata  
<Table length=1>
   key     val
  str10    str3
---------- ----
INSTRUMENT  ACS
>>> acsdq.parser.tab
<Table length=17>
DQFLAG ...                         LONG_DESCRIPTION
uint16 ...                              str74
------ ... ---------------------------------------------------------------
     0 ...                                                      Good pixel
     1 ...                                         Lost during compression
     2 ...                                          Replaced by fill value
     4 ... Bad detector pixel or beyond aperture or HRC upper-right defect
     8 ...              Masked by aperture feature or HRC occulting finger
    16 ...                                                       Hot pixel
   ... ...                                                             ...
   512 ...     Bad pixel in reference file (FLAT, polarizer, or dust mote)
  1024 ...                                                     Charge trap
  2048 ...                                          A-to-D saturated pixel
  4096 ...       Cosmic ray and detector artifact (AstroDrizzle, CR-SPLIT)
  8192 ...                                             Cosmic ray (ACSREJ)
 16384 ...                                        Manually flagged by user
 32768 ...                                                        Not used

Interpret DQ value for a single pixel at IRAF-style coordinate X=1457 and Y=170. Also display the original pixel value for comparison:

>>> acsdq.interpret_pixel(1457, 170)  
<Table length=3>
DQFLAG SHORT_DESCRIPTION LONG_DESCRIPTION
uint16        str9            str74
------ ----------------- ----------------
    16               HOT        Hot pixel
    32               CTE         CTE tail
  1024              TRAP      Charge trap
>>> acsdq.data[169, 1456]  
1072

This is the same as above, except pixel is given as 0-indexed values:

>>> acsdq.interpret_pixel(1456, 169, origin=0)

Interpret DQ values for all pixels. Then, extract mask for interpreted DQ value of 16 (hot pixel) and display pixel values for that mask:

>>> acsdq.interpret_all()  
Parsing DQ flag(s)...
Done!
Run time: 2.822 s
N_FLAGGED: 550656/8388608 (6.564%)
FLAG=1    : 0 (0.000%)
FLAG=2    : 0 (0.000%)
FLAG=4    : 0 (0.000%)
FLAG=8    : 0 (0.000%)
FLAG=16   : 78901 (0.941%)
FLAG=32   : 78915 (0.941%)
FLAG=64   : 343917 (4.100%)
FLAG=128  : 20373 (0.243%)
FLAG=256  : 15202 (0.181%)
FLAG=512  : 26623 (0.317%)
FLAG=1024 : 5128 (0.061%)
FLAG=2048 : 15117 (0.180%)
FLAG=4096 : 0 (0.000%)
FLAG=8192 : 0 (0.000%)
FLAG=16384: 0 (0.000%)
FLAG=32768: 0 (0.000%)
>>> hotmask = acsdq.dq_mask(16)  
>>> acsdq.data[hotmask]  
array([528, 528, 528, ..., 560, 560, 560], dtype=int16)

If you are only interested in a subset of the data, e.g., the 100x100 corner, you can also pass the subarray of interest into ImageDQ like this:

>>> from astropy.io import fits
>>> dqdata = fits.getdata('jbt7a3k7q_flc.fits', 3)  # DQ, 1  
>>> dqparser = DQParser.from_instrument('ACS')
>>> acsdq = ImageDQ(dqdata[:100, :100], dqparser=dqparser)  
>>> acsdq.interpret_all()  
Parsing DQ flag(s)...
Done!
Run time: 0.006 s
N_FLAGGED: 366/10000 (3.660%)
FLAG=1    : 0 (0.000%)
FLAG=2    : 0 (0.000%)
FLAG=4    : 0 (0.000%)
FLAG=8    : 0 (0.000%)
FLAG=16   : 111 (1.110%)
FLAG=32   : 21 (0.210%)
FLAG=64   : 115 (1.150%)
FLAG=128  : 0 (0.000%)
FLAG=256  : 0 (0.000%)
FLAG=512  : 100 (1.000%)
FLAG=1024 : 0 (0.000%)
FLAG=2048 : 0 (0.000%)
FLAG=4096 : 49 (0.490%)
FLAG=8192 : 0 (0.000%)
FLAG=16384: 0 (0.000%)
FLAG=32768: 0 (0.000%)
>>> acsdq.data.shape  
(100, 100)

Create a class instance for HST/WFC3 DQ parser (without image). Then, interpret a given DQ value:

>>> wfc3dq = DQParser.from_instrument('WFC3')
>>> wfc3dq.interpret_dqval(16658)
<Table length=4>
DQFLAG SHORT_DESCRIPTION                 LONG_DESCRIPTION
uint16        str9                            str69
------ ----------------- ------------------------------------------------
     2            FILLED                           Replaced by fill value
    16               HOT                                        Hot pixel
   256         SATURATED              Full-well or A-to-D saturated pixel
 16384             CRMAX Pixel has more than max CRs, ghost, or crosstalk
class reftools.interpretdq.DQParser(definition_file)

Class to handle parsing of DQ flags.

Definition Table

A “definition table” is an ASCII table that defines each DQ flag and its short and long descriptions. It can have optional comment line(s) for metadata, e.g.:

# INSTRUMENT = HSTGENERIC

It must have three columns:

  1. DQFLAG contains the flag value (uint16).

  2. SHORT_DESCRIPTION (string).

  3. LONG_DESCRIPTION (string).

Example file contents:

# INSTRUMENT = HSTGENERIC
DQFLAG SHORT_DESCRIPTION LONG_DESCRIPTION
0      "OK"              "Good pixel"
1      "LOST"            "Lost during compression"
...    ...               ...

The table format must be readable by astropy.io.ascii.

Parameters

definition_file (str) – ASCII table that defines the DQ flags (see above).

tab

Table object from given definition file.

Type

astropy.table.Table

metadata

Table object from file metadata.

Type

astropy.table.Table

classmethod from_instrument(instrument)

Use pre-defined DQ flags for the given instrument (HST only for now). The data files are distributed with this module.

Parameters

instrument ({'ACS', 'WFC3'}) – Currently not all instruments are supported. If the instrument is not found, a generic definition is used.

Returns

cls – An instance of this class.

Return type

DQParser

interpret_array(data, verbose=True)

Interpret DQ values for an array.

Warning

If the array is large and has a lot of flagged elements, this can be resource intensive.

Parameters

  • data (ndarray) – DQ values.

  • verbose (bool) – Print info to screen.

Returns

dqs_by_flag – Dictionary mapping each interpreted DQ value to indices of affected array elements.

Return type

dict

interpret_dqval(dqval)

Interpret DQ values for a single pixel.

Parameters

dqval (int) – DQ value.

Returns

dqs – Table object containing a list of interpreted DQ values and their meanings.

Return type

astropy.table.Table

class reftools.interpretdq.ImageDQ(data, dqparser=None)

Class to handle DQ flags in an image.

Parameters

  • data (ndarray) – DQ data array to interpret.

  • dqparser (DQParser or None) – DQ parser for interpretation. If not given, default is used.

data

Same as input.

Type

ndarray

parser

DQ parser for interpretation.

Type

DQParser

Raises

ValueError – Invalid image dimension.

dq_mask(dqval)

Generate mask for the given interpreted DQ value.

Parameters

dqval (int) – Interpreted DQ value.

Returns

mask – Boolean mask where the affected pixels are marked True.

Return type

ndarray

Raises

ValueError – Missing interpreter result or invalid DQ value.

classmethod from_fits(filename, ext=('DQ', 1), inskey='INSTRUME')

Use data from given FITS image and create DQ parser based on header value.

Parameters

  • filename (str) – Image filename.

  • ext (str or int or tuple) – Extension identifier, as accepted by astropy.io.fits.

  • inskey (str) – Keyword value in PRIMARY header that defines the instrument.

Returns

cls – An instance of this class.

Return type

ImageDQ

interpret_all(verbose=True)

Interpret DQ values for all flagged pixels. Results are stored in self._dqs_by_flag internal cache.

Warning

If the image is large and has a lot of flagged pixels, this can be resource intensive.

Parameters

verbose (bool) – Print info to screen.

interpret_pixel(x, y, origin=1)

Construct a user-friendly table object with interpreted DQ values for a given pixel location.

Parameters

  • x (int) – X and Y pixel coordinates. Also see origin.

  • y (int) – X and Y pixel coordinates. Also see origin.

  • origin ({0, 1}) – Indexing system for the pixel values given. Use 0 for Python 0-indexed system and 1 for IRAF 1-indexed system.

Returns

dqs – Table object containing a list of interpreted DQ values and their meanings.

Return type

astropy.table.Table

pixlist(origin=1)

Convert cached results to a list of (X, Y) pixel coordinates.

Parameters

origin ({0, 1}) – Indexing system for the pixel values returned. Use 0 for Python 0-indexed system and 1 for IRAF 1-indexed system.

Returns

pixlist_by_flag – Dictionary mapping each interpreted DQ value to a list of coordinates.

Return type

dict