Notes installing the SBEAMS - Cytometry module $Id: Cytometry.installnotes,v 1.0 2005/01/04 ------------------------------------------------------------------------------- 1) Software Dependencies You must first install the SBEAMS Core. See the separate installation notes (sbeams.installnotes) on how to accomplish that. The following Perl Modules often not found on a standard UNIX/Linux setup are required to successfully use SBEAMS - Cytometry (in addition to the dependencies for the SBEAMS Core). GD::GRAPH 2) Installation Location SBEAMS is designed to live entirely in the "htdocs" area of your Apache web server. For the remainder of this installation, it will be assumed that your installation is configured as follows; compensate for your specific setup: servername: db DocumentRoot: /local/www/html Primary location: directly located in DocumentRoot, /local/www/html/sbeams --> http:/db/sbeams/ Development location: In a dev1 tree starting in the DocumentRoot, /local/www/html/dev1/sbeams -> http:/db/dev1/sbeams/ All modules live in the same area and should be unpacked into the main SBEAMS area. ------------------------------------------------------------------------------- 3) Create and populate the database It is assumed that you have already created and tested your SBEAMS Core database. You may either create a separate database for the Cytometry or you can put everything in the same database. Note that some database engines (rare now) may not permit cross-database queries in which case your may NOT use separate databases. If you do use separate databases, you may not be able to enforce referencial integrity between tables in the different databases. This may or may not be a significant concern. - If you decide on a separate database, create it and within it, create users "sbeams" and "sbeamsro" as a read/write account and a read-only account, respectively, as done for the Core. - Generate the appropriate schema for your type(s) of database as follows: setenv SBEAMS /local/www/html/dev1/sbeams cd $SBEAMS/lib/scripts/Core foreach dbtype ( mssql mysql pgsql oracle ) ./generate_schema.pl \ --table_prop ../../conf/Cytometry/Cytometry_table_property.txt \ --table_col ../../conf/Cytometry/Cytometry_table_column.txt \ --schema_file ../../sql/Cytometry/Cytometry\ --destination_type $dbtype end - Verify that the SQL CREATE and DROP statements have been correctly generated in $SBEAMS/lib/sql/Cytometry/ - Execute the statements to create and populate the database with some bare bones data and indexes (for faster loading and querying): SQL Server Example: To CREATE and POPULATE: sqsh -i $SBEAMS/lib/sql/Cytometry/Cytometry_CREATETABLES.mssql -D sbeamsdev sqsh -i $SBEAMS/lib/sql/Cytometry/Cytometry_POPULATE.mssql -D sbeamsdev sqsh -i $SBEAMS/lib/sql/Cytometry/Cytometry_CREATECONSTRAINTS.mssql -D sbeamsdev To CREATE indexes: sqsh -i $SBEAMS/lib/sql/Cytometry/Cytometry_CREATEINDEXES.mssql -D sbeamsdev To DROP: sqsh -i $SBEAMS/lib/sql/Cytometry/Cytometry_DROPCONSTRAINTS.mssql -D sbeamsdev sqsh -i $SBEAMS/lib/sql/Cytometry/Cytometry_DROPTABLES.mssql -D sbeamsdev Note that the Cytometry_POPULATE.mssql is not autogenerated and should probably work for all flavors of database Examples for table creation for other database flavors can be found in the Core installation notes and will not be repeated here. ------------------------------------------------------------------------------- 4) Edit the SBEAMS Configuration files setenv SBEAMS /local/www/html/dev1/sbeams cd $SBEAMS/lib/conf edit SBEAMS.conf Specifically: set DBPREFIX{Cytometry} = Cytometry.dbo. (if you created a new Cytometry database) or DBPREFIX{Cytometry} = "core database prefix" (if you added the Cytometry tables to your core installed database)) ################################################################################## -loadFcsWeb.pl is used to load .fcs file from the web interface -loadFcs.pl is used to load .fcs files via the command line interface -$PHYSICAL_BASE_DIR/data/Cytometry. All data structures created by the analysis of the uploaded .fcs files are saved there and need to be global read and write. -$HTML_BASE_DIR/images/tmp Temporary image files and plots will be stored in $HTML_BASE_DIR/images/tmp. We installed a cron job for loadFcs.pl. It scans a directory recursively for .fcs files on a nightly basis. Due to this we created several log and watch files to keep an eye on loadFcs. Create a watch dir, edit $WATCHDIR to your dir in $SBEAMS/lib/scripts/Cytometry/loadFcs.pl First, loadFcs.pl checks for the presence of a watch file and aborts if one is found, since this would indicate that another instance of loadFcs.pl with the same start dir and project_id arguments is running or a previous run aborted prematurely. This also prevents trampling on each others data during long data uploads. The watch file is deleted at the end of a successfull upload Create a log file Edit $LOGFILE to your log file in $SBEAMS/lib/scripts/Cytometry/loadFcs.pl Create a log file directory if needed The log file traces the process of loadFcs.pl ####################################################################################+ ------------------------------------------------------------------------------- 5) Populate the driver tables cd $SBEAMS/lib/scripts/Core set CONFDIR = "../../conf" ./update_driver_tables.pl $CONFDIR/Cytometry/Cytometry_table_property.txt ./update_driver_tables.pl $CONFDIR/Cytometry/Cytometry_table_column.txt If this doesn't work. Do not proceed, debug first. ------------------------------------------------------------------------------- 6) Add groups Log in via the web interface as a user with Administrator privileges, switch to the Admin group using the pull-down menu at the top, and add two work groups: [SBEAMS Home] [Admin] [Manage Work Groups] [Add Work Group] Add entries for (exacly as shown!): Cytometry_user Cytometry_admin Cytometry_readonly (Note that after INSERTing the first, you can click [Back], edit the previous information slightly, and click [INSERT] to add another.) Cytometry_admin has privilege over all tables in the Cytometry module, while the Cytometry_user only has access to certain tables and may often not modify other users records. The Cytometry_readonly group is a separate group for looky loos. Now go to [Admin, Manage User Group Associations], [Add ...], and add yourself and whoever else to these groups as appropriate. Now set up the table group securities: rowprivate - Cytometry_user - data_writer rowprivate - Cytometry_admin - data_writer rowprivate - Cytometry_readonly - data_writer project - Cytometry_user - data_writer project - Cytometry_admin - data_modifier Cytometry_infrastructure - Cytometry_admin - data_modifier Cytometry_infrastructure - Cytometry_user - data_writer Now that the Cytometry driver tables are loaded, and the groups have been established, you should be able to go to the web site again and click on SBEAMS - Cytometry and explore the tables. They're all going to be empty, but you shouldn't get any errors, just empty resultsets. If this doesn't work. Do not proceed, debug first. ------------------------------------------------------------------------------- 7) Add some sample data (logged in as your user account) Add a Project as follows: - Switch to the Cytometry_user group by using the drop-down box at top - Click on [Cytometry Home] - Under "Projects You Own", click [Add A New Project]. Required fields are in red. If you don't have a budget number, enter NA. - Fill in the appropriate information and [INSERT] -Add some data to tissueType, sortEntity and sortType tables using the web interface (Sort Entity --> Add Sort Entity --> [INSERT]) -Add some sample FCS RUN as follows: - this can be done on the command line (more below) or through the web interface Web Interface (main.cgi sends command line options to loadWebFcs.pl) - Click on FCS Runs ---> Add Fcs Run - Fill in the required and optional information and click [INSERT] For our use we created '.fcs' file specifications according to the published Standard for Flow Cytometry FCS2.0 as described in Society for Analytical Cytology (Cytometry 11:323-332 (1990)) to parse relevant information from the header section of the .fcs file to populate the database (See $SBEAMS/doc/Cytometry/FCSHEADER.TXT). However, the values from the web form, if present, will override the values from the .fcs file. We also specified a Cytometry "sampleName" format which easily tracks the origin and history of a cytometry sample. This has also the advantage that the database is automatically updated with new sortEntities and sortTypes if neccessary. Click on "Sample Name Coding" in the left panel to see our specifications. The load process will not break if the sampleName or the info from the .fcs header differs from our specifications. Programmatically -use loadFcs.pl dirToScan project_id it would be good to add a "date dir" in the path of your dirToScan, this way .fcs files with the same names, but different date (yyyy-mm-dd), will not be overwritten (uncomment #DATE in loadFcs.pl) cd $SBEAMS/lib/scripts/Cytometry ./loadFcs.pl $SBEASMS/Cytometry/testData/ project_id This will scan the dirToScan and add new '.fcs' files to the database ################################################################################################################# ------------------------------------------------------------------------------- ------------------------------------------------------------------------------- Troubleshooting 1) How to re-generate and update table schemas. - Drop table and (firstly) its contraints, via SQL commands. Look under DOMAIN_DROPCONSTRAINTS.mssql and DOMAIN_DROPTABLES.mssql. e.g. look under Core_DROPCONSTRAINTS.mssql and Core_DROPTABLES.mssql - Edit the appropriate $DOMAIN_table_column.txt file e.g. make a field in conf/Core/Core_table_property.txt nullable=N - Generate new schema files using generate_schema.pl e.g. cd $SBEAMS/lib/scripts/Core ./generate_schema.pl --table_prop ../../conf/Core/Core_table_property.txt --table_col ../../conf/Core/Core_table_column.txt --schema_file ../../sql/Core/Core --destination_type mssql - Now re-create table and its contraints using the new (updated) dll. e.g. look under Core_CREATETABLES.mssql and Core_CREATECONSTRAINTS.mssql - Populate the table, if required. e.g. look in Core_POPULATE.mssql