From Helioviewer Wiki
The JP2Gen software converts solar FITS files to JP2 files for use with the Helioviewer project.
The source code can be downloaded from here - you will need the version control software Bazaar installed in order to download the latest stable trunk. You will also need IDL and Solarsoft with at least the following instrument packages :
gen hessi spex xray goes eit cds mdi lasco trace eis chianti stereo secchi euvi
Note that some of these packages require environment variables to be set (usually to find databases). Please consult the details for each instrument to correctly install the software.
You will also need access to the relevant FITS files. The current list of supported datasets can be found here. We explicitly support code to convert these FITS files to JP2 files that can be used with the Helioviewer Project.
The JP2Gen subdirectory /scripts contains Python code that requires the use of the PySQLite package. Python 2.5 and above contain this package as part of the standard distribution. If you have an older version of Python, you can download PySQLite from here.
1. Create a directory where you are going to store the JP2Gen programs, i.e., /home/jsmith/hv.
2. Create a directory where you are going to store the output from JP2Gen i.e., /home/jsmith/PUT_JP2_FILES_HERE.
3. Update your IDL path to include the location of the JP2Gen programs, i.e., in your .idl_startup file include the line !PATH = !PATH + ':' + EXPAND_PATH('+/home/jsmith/hv',/all_dirs).
5. Copy the file /home/jsmith/hv/jp2gen/local/EXAMPLE_hv_writtenby.pro into /home/jsmith/hv/jp2gen/local/hv_writtenby.pro . This file is where you will store your local default settings. Edit the file as indicated by the header information in the file.
6. Start up SSWIDL.
Transferring JPEG2000 files to a remote location
JP2Gen can push the JP2 files you have written to another computer. You need to edit the file local/hv_writtenby.pro to change the from and to computers, users and file directories as appropriate. There are two commands that may be used to do the transfer. The first one looks in the outgoing directory once only and transfer whatever is there
The second command repeats indefinitely transferring whatever is in the outgoing directory 15 minutes after the completion of the previous transfer.
The amount of time (in minutes) between consecutive searches of the outgoing directory can be changed.
JP2Gen can write out a simple webpage where the current status of all your quicklook processes can be monitored. You need to edit the file local/hv_writtenby.pro to point the structure tag "webpage" to your web space. The created webpage is called jp2gen_monitor.html , You can activate the monitoring system by using the JP2Gen command:
where <minutes> is the cadence (in minutes) for the re-creation of the webpage, i.e., this command will update the status of JP2Gen once every 15 minutes.
Helioviewer device setup files
Each instrument has a device setup file associated with it. This is actually a function which defines all the necessary parameters (JP2 compression factors for example) for that particular device. You can write your own if you want to use something other than the default choices, and you can call it through:
IDL> hv_eit_prep2jp2,date_start = 'yyyy/mm/dd',date_end = 'YYYY/MM/DD',details = "my_setup_file.pro"
JP2Gen will look for this file and it will expect to find an IDL function called "my_setup_file" that defines a single structure through which one can define all the parameters you need. If you need to add any extra functionality to write out the JPEG2000 files you want, please let us know. Full details on the structure that is defined in a Helioviewer device setup file can be found here - see step 4.
What JP2Gen does
For each supported dataset, JP2Gen uses standard SSW routines to do all the standard pre-processing and calibration to produce a useful science quality image. That image is then written as a JPEG2000 file. The routines are written to process batches of files, delimited by time. The structure of the JP2Gen programs and their output is as follows:
|$HV_JP2GEN/||subdirectory containing JP2GEN|
|$HV_JP2GEN/gen||general purpose routines called by all the device-specific routines.|
|$HV_JP2GEN/<mission>/<nickname>||the routines specific to a given device known as nickname on the mission/observatory "mission".|
|$HV_JP2GEN_WRITE/write||the directory where the output files, including JP2s, are stored.|
|$HV_JP2GEN_WRITE/write/outgoing/<nickname>/<directory structure>||the outgoing directory. Files and directory structure in here are put here to be copied to remote locations.|
|$HV_JP2GEN_WRITE/write/web||web notices directory. This directory stores text files that are picked up by other routines to create a web page which monitors the state of JP2Gen processes.|
|$HV_JP2GEN_WRITE/write/vX.YY||version X.YY of the JP2 files and related output.|
|$HV_JP2GEN_WRITE/write/vX.YY/jp2||All the JP2 files written by JP2Gen.|
|$HV_JP2GEN_WRITE/write/vX.YY/jp2/<nickname>||JP2 files for the device known as nickname.|
|$HV_JP2GEN_WRITE/write/vX.YY/jp2/<nickname>/<directory structure>||the JPEG2000 files stored in the correct directory structure.|
|$HV_JP2GEN_WRITE/write/vX.YY/log||All the log files written by JP2Gen.|
|$HV_JP2GEN_WRITE/write/vX.YY/log/<nickname>||log files concerning the conversion of FITS to JP2 for the device known as nickname.|
|$HV_JP2GEN_WRITE/write/vX.YY/log/<nickname>/<directory structure>||Log files for each JP2 file written.|
|$HV_JP2GEN_WRITE/write/vX.YY/db||All the database files written by JP2Gen.|
|$HV_JP2GEN_WRITE/write/vX.YY/db/<nickname>||database files concerning the conversion of FITS to JP2 for the device known as nickname.|
|$HV_JP2GEN_WRITE/write/vX.YY/db/<nickname>/<directory structure>||A database file for the files written with <nickname> and <measurement>.|
The nickname is the common everyday name used in conversation for a particular device.
JP2 file design: decisions and reasons
The Helioviewer project has been exclusively focused on using an archive of JPEG2000 formatted images to deliver solar physics image data over the web since September 2008. Since then, the design of the JPEG2000 files for use with the Helioviewer has undergone many revisions. The design of these files, and the reason for the choices we made are described here. This page will be updated as reviews of the functionality of the Helioviewer Project make different demands on the design of our JPEG2000 files.
The Helioviewer Project supplies IDL code that converts FITS data from a range of instruments into JP2 files. The current list of instruments, and the time-range of data for which JP2 files are available, is given here.
Adding new datasets to the Helioviewer Project
You want to have your data written into JP2 files for use with the Helioviewer Project - how do you do that? Full information can be found here.
The clients need to know key properties of the original FITS files. When we convert a FITS file to a JP2 file, we copy the FITS keywords as XML keywords and store them in an XML box at the top of the JP2 file. This meta-data is accessed by various components of the Helioviewer Project. A key describing the currently used FITS header keywords is maintained on the following page: FITS to HV conversion key.
FITS to JP2 Benchmarking
We have performed some benchmarking concerning how long it takes to read a FITS file, prep it using IDL SSW routines, and write it to disk. Current benchmarks can be found here.
Running FITS to JP2 processes remotely
It is possible to run the conversion processes when logged into the conversion machine from other systems. It basically involves running SSWIDL as a batch process with a re-direct of the text output to a file. You need to write a short 'batch file' in legal IDL code that gets piped into SSWIDL
[computer:] nohup $SSW/gen/setup/ssw_idl < batchfile.pro >& batchfile.output.txt &
The "nohup" command runs the process without being attached to a terminal. This means the process can run when you log out of the machine running the conversion software. Since many of the JP2Gen programs generate text output, this has to be redirected somewhere (in this case, a log file of the output). This command was based on code taken from http://exoplanets.sfsu.edu/lab/idl_background_jobs.txt .
Running the transfer script processes remotely
Often the machine producing the JPEG2000 files is not the one that actually serves those files. Therefore, the files have to be transferred from one machine to another. In JP2Gen, this is achieved by using rsync which is called by hv_jp2_transfer.pro. The rsync command connects to the JPEG2000 serving machine using ssh.
The solution for running the JP2 file creation processes in the background does not work for the transfer process. This is because the ssh connection gets refused by the remote machine when run under nohup. Somehow, the ssh process is aware that you have logged out and the process is running in the background.
The solution is to run the transfer process using the "byobu" package. What is this? Well, from its webpage:
"Byobu is a Japanese term for decorative, multi-panel screens that serve as folding room dividers. As an open source project, Byobu is an elegant enhancement of the otherwise functional, plain, practical GNU Screen. Byobu includes an enhanced profile and configuration utilities for the GNU screen window manager, such as toggle-able system status notifications."
which naturally leads to the question, what is GNU Screen?, and from its webpage:
"Screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells. Each virtual terminal provides the functions of the DEC VT100 terminal and, in addition, several control functions from the ANSI X3.64 (ISO 6429) and ISO 2022 standards (e.g., insert/delete line and support for multiple character sets). There is a scrollback history buffer for each virtual terminal and a copy-and-paste mechanism that allows the user to move text regions between windows. When screen is called, it creates a single window with a shell in it (or the specified command) and then gets out of your way so that you can use the program as you normally would. Then, at any time, you can create new (full-screen) windows with other programs in them (including more shells), kill the current window, view a list of the active windows, turn output logging on and off, copy text between windows, view the scrollback history, switch between windows, etc. All windows run their programs completely independent of each other. Programs continue to run when their window is currently not visible and even when the whole screen session is detached from the users terminal."]:
Running byobu allows you to keep the ssh process running even when you log out. When using byobu, the transfer process command is:
[computer:] $SSW/gen/setup/ssw_idl < batch_transfer.pro >& batch_transfer.output.txt &
You may have to install screen and byobu on your systems in order to use this solution.
JP2 File Versions
Version v0.8 of the JP2s released 2010-03-09. Simple database output written every day for every nickname/measurement combination. This aids with the transfer of files from one place to another, and also means we can keep track of which input file was converted into which JP2 file.
Version v0.7 of the JP2s released 2010-02-24. Output directory structure now <nickname>/<measurement>/YYYY/MM/DD. Each JP2 file gets an individual log file. Device setup files provided for each device - use can use default file provided by the Helioviewer Project or write their own. JP2 files themselves contain two new HV_ tags describing the setup file, but are otherwise unchanged.
Version v0.6 of the JP2s released 2009-11-19. Fixes errors in EIT file creation. Sub-sampled images 512 x 512 are no longer resampled up to 1024 x 1024, Subsampled EIT FITS files remain at 512 x 512 pixels and have a correspondingly larger number of arcseconds per pixel.
Version v0.5 of the JP2s released 2009-11-16. This set removes the LASCO transparency mask. Also, introduced version numbers NOT based on dates. New version numbers and Bazaar revision number (of JP2GEN) now introduced via keywords into the JP2 XML headers.
Version v20090917 (released 2009-09-17) is the first version that removes the image rescaling/recentering done before writing to JP2. These trasnformations are now done via the client.
Version v20090511 (released 2009-05-11)written to demonstrate client capabilities at the Solar Physics Division Meeting in Boulder, CO, June 14-18, 2009, (http://spd2009.boulder.swri.edu/).
- JPEG2000 file design for the Helioviewer Project - reasons why we designed the JP2 FITS files the way we did.
- FITS2HV - a description of the FITS header tags and their meaning in the Helioviewer Project.
- Helioviewer JP2 XML Tags - tags that the Helioviewer Project adds to each JP2 file
- JPEG 2000 - more detail on the JPEG2000 standard.