Notes installing the SBEAMS - Cytometry module

$Id: Cytometry.installnotes,v 1.0 2005/01/04


-------------------------------------------------------------------------------
1) Software Dependencies

You must first install the SBEAMS Core.  See the separate installation
notes (sbeams.installnotes) on how to accomplish that.

The following Perl Modules often not found on a standard UNIX/Linux
setup are required to successfully use SBEAMS - Cytometry (in addition
to the dependencies for the SBEAMS Core).

GD::GRAPH

2) Installation Location

SBEAMS is designed to live entirely in the "htdocs" area of your Apache
web server.  For the remainder of this installation, it will be assumed
that your installation is configured as follows; compensate for your
specific setup:
  servername: db
  DocumentRoot: /local/www/html
  Primary location: directly located in DocumentRoot,
                   /local/www/html/sbeams  --> http:/db/sbeams/
  Development location: In a dev1 tree starting in the DocumentRoot,
                   /local/www/html/dev1/sbeams  -> http:/db/dev1/sbeams/

All modules live in the same area and should be unpacked into the
main SBEAMS area.


-------------------------------------------------------------------------------
3) Create and populate the database

It is assumed that you have already created and tested your SBEAMS Core
database.  You may either create a separate database for the Cytometry 
or you can put everything in the same database.

Note that some database engines (rare now) may not permit
cross-database queries in which case your may NOT use separate
databases.  If you do use separate databases, you may not be able to
enforce referencial integrity between tables in the different
databases.  This may or may not be a significant concern.

- If you decide on a separate database, create it and within it,
  create users "sbeams" and "sbeamsro" as a read/write
  account and a read-only account, respectively, as done for the Core.

- Generate the appropriate schema for your type(s) of database as follows:

setenv SBEAMS /local/www/html/dev1/sbeams
cd $SBEAMS/lib/scripts/Core

foreach dbtype ( mssql mysql pgsql oracle )
  ./generate_schema.pl \
    --table_prop ../../conf/Cytometry/Cytometry_table_property.txt \
    --table_col ../../conf/Cytometry/Cytometry_table_column.txt \
    --schema_file ../../sql/Cytometry/Cytometry\
    
    --destination_type $dbtype
end


- Verify that the SQL CREATE and DROP statements have been correctly
  generated in $SBEAMS/lib/sql/Cytometry/

- Execute the statements to create and populate the database with some
  bare bones data and indexes (for faster loading and querying):

SQL Server Example:

To CREATE and POPULATE:
sqsh -i $SBEAMS/lib/sql/Cytometry/Cytometry_CREATETABLES.mssql -D sbeamsdev
sqsh -i $SBEAMS/lib/sql/Cytometry/Cytometry_POPULATE.mssql -D sbeamsdev
sqsh -i $SBEAMS/lib/sql/Cytometry/Cytometry_CREATECONSTRAINTS.mssql -D sbeamsdev

To CREATE indexes:
sqsh -i $SBEAMS/lib/sql/Cytometry/Cytometry_CREATEINDEXES.mssql -D sbeamsdev

To DROP:
sqsh -i $SBEAMS/lib/sql/Cytometry/Cytometry_DROPCONSTRAINTS.mssql -D sbeamsdev
sqsh -i $SBEAMS/lib/sql/Cytometry/Cytometry_DROPTABLES.mssql -D sbeamsdev

Note that the Cytometry_POPULATE.mssql is not autogenerated and should
probably work for all flavors of database

Examples for table creation for other database flavors can be found in the
Core installation notes and will not be repeated here.


-------------------------------------------------------------------------------
4) Edit the SBEAMS Configuration files

setenv SBEAMS /local/www/html/dev1/sbeams

cd $SBEAMS/lib/conf
edit SBEAMS.conf

Specifically:
set  
DBPREFIX{Cytometry}    = Cytometry.dbo. (if you created a new Cytometry database)
or 
DBPREFIX{Cytometry}    = "core database prefix"  (if you added the Cytometry tables to your core installed database))

##################################################################################

-loadFcsWeb.pl
    is used to load .fcs file from the web interface
-loadFcs.pl
   is used to load .fcs files via the command line interface

-$PHYSICAL_BASE_DIR/data/Cytometry.
     All data structures created by the analysis of the uploaded .fcs files are saved there and need to be global read and write.

-$HTML_BASE_DIR/images/tmp
    Temporary image files and plots will be stored in $HTML_BASE_DIR/images/tmp. 

We installed a cron job for loadFcs.pl. It  scans a directory recursively  for .fcs files on a nightly basis. Due to this 
we created several log and watch files to keep an eye on loadFcs.
Create a watch dir, edit $WATCHDIR to your dir in $SBEAMS/lib/scripts/Cytometry/loadFcs.pl
    First, loadFcs.pl  checks for the presence of a watch file and aborts if one is found, since this would
    indicate that another instance of loadFcs.pl with the same start dir and project_id arguments  is
    running or a previous run aborted prematurely. This also prevents trampling on each others data during long data uploads.
    The watch file is deleted at the end of a successfull upload
Create a log file
    Edit $LOGFILE to your log file in  $SBEAMS/lib/scripts/Cytometry/loadFcs.pl
    Create a log file directory if needed
    The log file  traces the process of loadFcs.pl    


####################################################################################+


-------------------------------------------------------------------------------
5) Populate the driver tables

cd $SBEAMS/lib/scripts/Core
set CONFDIR =  "../../conf"
./update_driver_tables.pl $CONFDIR/Cytometry/Cytometry_table_property.txt
./update_driver_tables.pl $CONFDIR/Cytometry/Cytometry_table_column.txt

If this doesn't work.  Do not proceed, debug first.


-------------------------------------------------------------------------------
6) Add groups

Log in via the web interface as a user with Administrator privileges,
switch to the Admin group using the pull-down menu at the top, and add
two work groups:
[SBEAMS Home] [Admin] [Manage Work Groups] [Add Work Group]
Add entries for (exacly as shown!):
  Cytometry_user
  Cytometry_admin
  Cytometry_readonly
(Note that after INSERTing the first, you can click [Back], edit the previous
information slightly, and click [INSERT] to add another.)

Cytometry_admin has privilege over all tables in the Cytometry
module, while the Cytometry_user only has access to certain tables
and may often not modify other users records.  The Cytometry_readonly
group is a separate group for looky loos.

Now go to [Admin, Manage User Group Associations], [Add ...], and add
yourself and whoever else to these groups as appropriate.

Now set up the table group securities:
  rowprivate - Cytometry_user - data_writer
  rowprivate - Cytometry_admin - data_writer
  rowprivate - Cytometry_readonly - data_writer
  project - Cytometry_user - data_writer
  project - Cytometry_admin - data_modifier
  Cytometry_infrastructure - Cytometry_admin - data_modifier
  Cytometry_infrastructure - Cytometry_user - data_writer



Now that the Cytometry driver tables are loaded, and the groups have been
established, you should be able to go to the web site again and click on
SBEAMS - Cytometry and explore the tables.  They're all going to be empty,
but you shouldn't get any errors, just empty resultsets.

If this doesn't work.  Do not proceed, debug first.


-------------------------------------------------------------------------------
7) Add some sample data

(logged in as your user account)

Add a Project as follows:

- Switch to the Cytometry_user group by using the drop-down box at top
- Click on [Cytometry Home]
- Under "Projects You Own", click [Add A New Project].  Required
  fields are in red.  If you don't have a budget number, enter NA.
- Fill in the appropriate information and [INSERT]

-Add some data to tissueType, sortEntity and sortType tables
 using the web interface (Sort Entity --> Add Sort Entity --> [INSERT])

-Add some sample  FCS RUN as follows:
- this can be done on the command line (more below)  or through the web interface
   Web Interface (main.cgi sends command line options to  loadWebFcs.pl)
- Click on FCS Runs --->  Add Fcs Run
- Fill in the required and optional information and click [INSERT] 
     For our use we created  '.fcs' file specifications according to the published 
     Standard for Flow Cytometry FCS2.0 as described in  Society for Analytical Cytology  (Cytometry 11:323-332 (1990))
     to parse relevant information from the header section of the .fcs file to populate the database
     (See $SBEAMS/doc/Cytometry/FCSHEADER.TXT).
     However, the values from the web form, if present,  will override the values from the .fcs file. 
     We also specified a Cytometry "sampleName"  format which easily tracks  the origin and history of a cytometry sample. This has
     also the advantage  that the database is automatically updated with new sortEntities and sortTypes if neccessary. 
     Click on "Sample Name Coding" in the left panel to see our specifications.
     
     The load process will not break if the sampleName or the info from the .fcs header differs from our specifications.

Programmatically 
-use loadFcs.pl dirToScan project_id
   it would be good to add a "date dir" in the path of  your  dirToScan, this way .fcs files with the 
   same names, but different date (yyyy-mm-dd), will not be overwritten (uncomment #DATE in loadFcs.pl)
   cd $SBEAMS/lib/scripts/Cytometry  
    ./loadFcs.pl $SBEASMS/Cytometry/testData/  project_id 
    This will scan the dirToScan and add new '.fcs' files to the database   
    


#################################################################################################################
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Troubleshooting

1) How to re-generate and update table schemas.

- Drop table and (firstly) its contraints, via SQL commands. Look under 
  DOMAIN_DROPCONSTRAINTS.mssql and DOMAIN_DROPTABLES.mssql.
  e.g. look under Core_DROPCONSTRAINTS.mssql and Core_DROPTABLES.mssql

- Edit the appropriate $DOMAIN_table_column.txt file
  e.g. make a field in conf/Core/Core_table_property.txt nullable=N

- Generate new schema files using generate_schema.pl
  e.g.  cd $SBEAMS/lib/scripts/Core
	./generate_schema.pl --table_prop ../../conf/Core/Core_table_property.txt --table_col ../../conf/Core/Core_table_column.txt --schema_file ../../sql/Core/Core --destination_type mssql

- Now re-create table and its contraints using the new (updated) dll.
  e.g. look under Core_CREATETABLES.mssql and Core_CREATECONSTRAINTS.mssql

- Populate the table, if required.
  e.g. look in Core_POPULATE.mssql




