astropy:docs

ImageLog API Reference

class moastro.imagelog.ImageLog(dbname, cname, server=None, url='localhost', port=27017)[source]

Base class for all mongodb-based image logs.

The intent is for this base class to access the entire collection of image records for a given project. Subclasses can be used that cater to specific parts of the data set (such as one instrument or another; one observing mode or another). This specificity is created with the queryMask, which is a query that will always be appending to the user’s query to ensure that only the requested types of data are returned. For example, this can be a query to only accept ‘MegaPrime’ under the INSTRUME key.

Parameters:

dbname : str

Name of MongoDB database.

cname : str

Name of MongoDB collection.

server : str

Name of the MongoDB server, as specified in ~/.moastro.json. If None, then the values of url and port will be adopted instead.

url : str

URL of MongoDB server.

port : int

Port of MongoDB server.

compress_fits(pathKey, selector={}, candidateImages=None, alg='Rice', q=4, delete=False)[source]
Parameters:alg – Compression algorithm. Any of:
  • Rice
  • gzip
Parameters:
  • q – quantization for floating-point images.
  • delete – set to true if the un-compressed original should be deleted
decompress_fits(pathKey, decompKey=None, decompDir=None, selector={}, delete=False, overwrite=False, nthreads=8)[source]

Decompresses FITS files at pathKey.

Parameters:
  • pathKey – field where FITS paths are found
  • decompKey – (optional) can be set to a field where the decompressed file can be found. Otherwise, the decompressed file path is written to pathKey.
  • selector – (optional) search criteria dictionary
  • delete – set to True to delete the compressed file.
  • overwrite – if False, then funpack will skip any files that have already been decompressed. That is, the output file must be present and recorded in the image log under decompPathKey.
  • nthreads – set to the number of threads. Multiprocessing is used if nthreads > 1.
delete_field(dataKey, selector=None, multi=True)[source]

Deletes a field from selected image records.

Parameters:
  • dataKey – field to be deleted.
  • selector – (optional) search selector dictionary
delete_files(pathKey, selector=None)[source]

Deletes all files stored under pathKey, and the reference in the image log.

Parameters:
  • pathKey – data key for path. Can include dot syntax.
  • selector – (optional) MongoDB query dictionay.
distinct(field, selector, images=None)[source]

Return the set of distinct values a field takes over the selection.

find(selector, images=None, one=False, **mdbArgs)[source]

Wrapper around MongoDB find().

find_dict(selector, images=None, fields=None)[source]

Analogous to find(), but formats the returned cursor into a dictionary.

find_images(selector, images=None)[source]

Get images keys that match the specified selector using MongoDB queries.

find_unique(dataKey, selector={}, candidateImages=None)[source]

Get the set of unique values of data key for images that meet the specified selector.

Note

This doesn’t actually use the distinct aggregation commmand in PyMongo, since it doesn’t yet support queries itself.

Parameters:
  • dataKey – data field whose values will compiled into a set of unique values.
  • selector – (optional) dictionary for data keys: data values that specifies what image keys should be returned. Any mongodb search dictionary will work.
  • candidateImages – (optional) a list of images to draw from; only images within the candidateImages set will be considered

Deprecated since version Use: distinct() instead.

get(selector, dataKeys, candidateImages=None)[source]

Get a dictionary of image key: {data key: data value, ...} for images that match the search selector. The returned dictionary contains data only with the requested dataKeys (i.e., a subset of the data base and records for each image are returned.)

Parameters:
  • selector – dictionary for data keys: data values that specifies what image keys should be returned. Any mongodb search dictionary will work.
  • dataKeys – data for each image that should be returned for the selected images.
  • candidateImages – (optional) a list of images to draw from; only images within the candidateImages set will be considered

Deprecated since version Use: find_dict() instead.

get_images(imageKeys, dataKeys)[source]

Same as get(), but operates on a sequence of image keys, rather than a search selector.

Deprecated since version Use: find_dict() instead.

getiter(selector, dataKeys, candidateImages=None)[source]

Returns a cursor to iterate through image records that meet the given selector. Each record is a dictionary, with the image key stored under _id.

Parameters:
  • selector – dictionary for data keys: data values that specifies what image keys should be returned. Any mongodb search dictionary will work.
  • dataKeys – data key(s) for each image that should be returned for the selected images. Can be string or a sequence of strings.
  • candidateImages – (optional) a list of images to draw from; only images within the candidateImages set will be considered

Deprecated since version Use: find() instead.

move_files(pathKey, newDir, selector=None, copy=False)[source]

Moves a file whose path is found under pathKey to the newDir directory. The directory is created if necessary. Old files are overwritten if necessary.

Parameters:
  • dataKey – field of path for files to be moved
  • newDir – directory where files should be moved to.
  • selector – (optional) search selector dictionary
print_rec(imageKey)[source]

Pretty-prints the record of imageKey

rename_field(dataKeyOld, dataKeyNew, selector=None, multi=True)[source]

Renames a field for all image log records found with the optional selector.

Parameters:
  • dataKeyOld – original field name
  • dataKeyNew – new field name
  • selector – (optional) search selector dictionary
search(selector, candidateImages=None)[source]

Get images keys that match the specified selector using MongoDB queries.

Parameters:
  • selector – dictionary for data keys: data values that specifies what image keys should be returned. Any mongodb search dictionary will work.
  • candidateImages – (optional) a list of images to draw from; only images within the candidateImages set will be considered
Returns:

the list of image keys for images that match the selector

dictionary.

Deprecated since version Use: find_images() instead.

set(imageKey, key, value, ext=None)[source]

Updates an image record by setting the key field to the given value.

set_frames(key, data)[source]

Does an update of data into the key field for data of an arbitrary collection of detectors.

Parameters:data – a dictionary of frame: datum, where frame is a tuple of (imageKey, ext)
class moastro.imagelog.MEFImporter(dbname, cname, server=None, url='localhost', port=27017)[source]

Base class for importing MEF (multi-extension FITS) into an imagelog.

The user subclass that inherits MEFImporter should do several things:

  1. Set the exts attribute to be a list of FITS extension integers to import. If the sole FITS image is in the PrimaryHDU, then leave this as an empty list. Otherwise, e.g., for WIRCam with images in four extension HDUs, set self.exts = [1, 2, 3, 4].
  2. Set or extend the copy_keys attribute to be a list of FITS header keys to import from the Primary HDU into the base image document.
  3. Set or extend the copy_ext_keys attribute to be a list of FITS header keys to import from each image extension.
  4. Implement generate_id to generate a document _id string.
  5. (optional) Implement post_base_ingest to modify the base image log document (a dict) after the header keys have been imported. This can be useful to add additional metadata, or to change the metadata schema from that in the FITS header.
  6. (optional) Implement post_ext_ingest to modify the document for each image extension after the extension header keys are imported.
Parameters:

dbname : str

Name of MongoDB database.

cname : str

Name of MongoDB collection.

server : str

Name of the MongoDB server, as specified in ~/.moastro.json. If None, then the values of url and port will be adopted instead.

url : str

URL of MongoDB server.

port : int

Port of MongoDB server.

static all_files(root, pattern, single_level=False)[source]

Yield file paths matching a pattern.

Adapted from Python Cookbook, 2nd Ed. 2.16.

static chip_footprint_polygon(header, hdulist)[source]

Create a Mongo-compatible polygon representing the chip footprint in equatorial cordinates. The polygon is a length-4 list, populated with length-2 lists of RA, Dec vertices.

generate_id(path, header)[source]

Generate the object id for this image.

Should be implemented by user. Raises NotImplementedError otherwise.

ingest(base_dir, suffix='.fits', recursive=True, preview=False)[source]

Runs the import pipeline.

Parameters:

base_dir : str

Directory where FITS files can be found.

suffix : str

Suffix of files to import. That is, all paths ending with this suffix are imported. This import fits files by default. But change to e.g., 's.fits' to get processed CFHT images, or '.fits.fz' to look for encrypted FITS images.

recursive : bool

If True, then the pipeline walks through directories contained in the base directory, looking for FITS files.

preview : bool

If True then the documents are not inserted into MongoDB, but only printed. Useful for debugging the ingest.

ingest_one(path, preview=False)[source]

Ingest a single FITS file at path.

Parameters:

path : str

Path to the FITS image.

preview : bool

If True then the documents are not inserted into MongoDB, but only printed. Useful for debugging the ingest.

post_base_ingest(doc, path, header)[source]

Hook for modifying document after ingesting the base FITS header.

This method can be implemented by the user to add additional data to the base image log document. Simply add data to to the dict doc.

post_ext_ingest(doc, header)[source]

Hook for modifying document after ingesting an extension header.

This method can be implemented by the user to add additional data to the extension-specific image log document. Simply add data to to the dict doc.

Page Contents