/goldenPath/help/bigRmskTrackDescExample.html:Genome_Browser_bigRmsk_RepeatMasker_Description	 
/goldenPath/help/interact.html:interact_and_bigInteract_Track_Format	Interact and bigInteract Track Format The interact (and bigInteract) track format displays pairwise interactions as arcs or half-rectangles connecting two genomic regions on the same chromosome. Cross-chromosomal interactions can also be represented in this format; the display shows the region on the currently viewed chromosome, with a vertical bar, labeled with the chromosome of the connected region (space permitting). For directional interactions such as SNP/gene, the interactions in the reverse direction are displayed as a dashed line or curve. An alternative 'cluster' view can also be configured for directional interaction data. This view groups interactions by source or by target, producing a display of linked blocks. This format is useful for displaying functional element interactions such as SNP/gene interactions, and is also suitable for low-density chromatin interactions, such as ChIA-PET, and other use cases with a limited number of interactions on the genome. It is not suitable for high-density chromatin data such as Hi-C. The interact format is available as a standalone plain text bed5+13 format for use with smaller datasets as a custom track, and as a binary indexed format (bigInteract) suitable for track hubs and custom tracks. The bigInteract format provides more track customization features (i.e. schema customization), and is recommended for users who can use command-line tools and have web-accessible data storage. If you do not have web-accessible data storage, please see the Hosting section of the Track Hub Help documentation. Interact format files are converted to bigInteract files using the program bedToBigBed, run with the -as option to pull in a special autoSql (.as) schema file defining the fields of the bigInteract. Interact format definition The following autoSql definition illustrates the basic schema supporting interact (and bigInteract) tracks. table interact "interaction between two regions" ( string chrom; "Chromosome (or contig, scaffold, etc.). For interchromosomal, use 2 records" uint chromStart; "Start position of lower region. For interchromosomal, set to chromStart of this region" uint chromEnd; "End position of upper region. For interchromosomal, set to chromEnd of this region" string name; "Name of item, for display. Usually 'sourceName/targetName/exp' or empty" uint score; "Score (0-1000)" double value; "Strength of interaction or other data value. Typically basis for score" string exp; "Experiment name (metadata for filtering). Use . if not applicable" string color; "Item color. Specified as r,g,b or hexadecimal #RRGGBB or html color name, as in //www.w3.org/TR/css3-color/#html4. Use 0 and spectrum setting to shade by score" string sourceChrom; "Chromosome of source region (directional) or lower region. For non-directional interchromosomal, chrom of this region." uint sourceStart; "Start position in chromosome of source/lower/this region" uint sourceEnd; "End position in chromosome of source/lower/this region" string sourceName; "Identifier of source/lower/this region" string sourceStrand; "Orientation of source/lower/this region: + or -. Use . if not applicable" string targetChrom; "Chromosome of target region (directional) or upper region. For non-directional interchromosomal, chrom of other region" uint targetStart; "Start position in chromosome of target/upper/this region" uint targetEnd; "End position in chromosome of target/upper/this region" string targetName; "Identifier of target/upper/this region" string targetStrand; "Orientation of target/upper/this region: + or -. Use . if not applicable" ) Column Explanations The first 5 fields of the interact format are the same as the first 5 fields of the standard BED format. See a graphical depiction below of the columns. When creating bigInteract files, we encourage you to customize the title and field descriptions of the prototype autoSql schema to better describe your data. Customizing this file will make your data more easily interpreted by users, who will see the field descriptions when accessing the track data from the Table Browser, when viewing items on the Genome Browser details pages (via the "view table schema" link), and (for users who download files), from the -as option of the bigBedInfo tool. As an example, if the dataset represents SNP/gene interactions, replace 'sourceName' and related fields with 'snpName', etc, and 'targetName' and related fields with 'geneName', etc., editing the field descriptions to reflect the changes you make. For non-directional data such as ChIA-PET, one could use 'region1' and 'region2'. As the browser display of this format only shows the paired region labels on mouseover, we recommend including a BED or other format file to display the source and target region labels. Creating interact and bigInteract custom tracks Example #1 In this example, you will create an interact custom track using example SNP/gene interaction data in multiple tissues. This example uses the interaction coloring and directionality features of the interact track type. 1. Paste the following track line into the custom track management page for the human assembly hg19. track type=interact name="interact Example One" description="An interact file" interactDirectional=true maxHeightPixels=200:100:50 visibility=full browser position chr12:40,560,500-40,660,499 #chrom chromStart chromEnd name score value exp color sourceChrom sourceStart sourceEnd sourceName sourceStrand targetChrom targetStart targetEnd targetName targetStrand chr12 40572709 40618813 rs7974522/LRRK2/muscleSkeletal 0 0.624 muscleSkeletal #7A67EE chr12 40572709 40572710 rs7974522 . chr12 40618812 40618813 LRRK2 + chr12 40579899 40618813 rs17461492/LRRK2/muscleSkeletal 0 0.624 muscleSkeletal #7A67EE chr12 40579899 40579900 rs17461492 . chr12 40618812 40618813 LRRK2 + chr12 40614433 40618813 rs76904798/LRRK2/nerveTibial 0 0.625 nerveTibial #FFD700 chr12 40614433 40614434 rs76904798 . chr12 40618812 40618813 LRRK2 + chr12 40618812 40652520 rs2723264/LRRK2/lung 0 1.839 lung #9ACD32 chr12 40652519 40652520 rs2723264 . chr12 40618812 40618813 LRRK2 + 2. Click the "submit" button. After the file loads in the Genome Browser, you should see four interactions displayed; four variants interacting with the same gene (LRRK2). Hovering the mouse over the curve peak of an interaction will display the interaction name (SNP/gene/tissue). Hovering over an interaction end will show the name of the end region (e.g. SNP or gene). Clicking at one of the hoverable regions will show the details page for the interaction. The interactDirectional setting causes reverse direction interactions (where target precedes source) to be displayed as dashed lines. In this example, the green (lung) interaction is in the reverse direction. Example #2 In this example, you will create an interact custom track using example chromatin interaction data. This type of data is non-directional and commonly would represent a single experiment, with the interaction score being of interest. The settings below display using the gray-scale coloring feature, where the darkness of the interaction is based on the score. 1. Paste the following track line into the custom track management page for the human assembly hg19. track type=interact name="interact Example Two" description="Chromatin interactions" useScore=on maxHeightPixels=200:100:50 visibility=full browser position chr3:64,562,440-64,642,288 chr3 64496901 64584378 . 375 3 . 0 chr3 64496901 64498901 . . chr3 64581378 64584378 . . chr3 64568052 64569134 . 400 3 . 0 chr20 52552477 52556062 . . chr3 64568052 64569134 . . chr3 64596901 64615378 . 175 3 . 0 chr3 64596901 64600855 . . chr3 64612677 64615378 . . chr3 64623042 64636663 . 800 4 . 0 chr3 64623042 64625153 . . chr3 64632961 64636663 . . 2. Click the "submit" button. After the file loads in the Genome Browser, you should see four interactions displayed on chromosome 3. Two of the interactions have both interacting regions in the browser view, and two have a single region. One of these interacts across chromosomes (with a region on chromosome 20), and the other with a region outside of the browser window (indicated by rectangular connector). The darkness of the interaction indicates the strength of the interaction. Also see the graphical representation of each column of this example data below. Example #3 In this example, you will create a bigInteract track out of an existing bigInteract format file, located on the UCSC Genome Browser http server. This file contains data for the hg19 assembly. This example also contains the interactUp=true setting to flip the arcs of the interact display. To create a custom track using this file: 1. Construct a track line referencing the file and set the browser position to show region of interest in the file: track type=bigInteract interactUp=true name="interact Example Three" description="A bigInteract file" useScore=on visibility=full bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/interact/interactExample3.inter.bb browser position chr3:63,820,967-63,880,091 2. Paste the track line into the custom track management page for the human assembly hg19. 3. Click the "submit" button. After the file loads in the Genome Browser, you should see a number of interactions, all arching as hills instead of valleys, with some curved and many rectangular indicating a connector to a region outside of the browser window. Press the 10x zoom out button to see the full connections. Example #4 In this example, you will use an example BED file to create a bigInteract file, allowing the data to be remotely accessed and exist within a track hub. The track settings for bigInteract on a hub can be viewed here. 1. Download the example file here. 2. Download the fetchChromSizes and bedToBigBed programs from the utilities directory appropriate to your operating system. 3. Use fetchChromSizes to create a chrom.sizes file for the UCSC database you are working with (hg19 for these examples). Alternatively, you can download the chrom.sizes file for any assembly hosted at UCSC from our downloads page (click on "Full data set" for any assembly). For example, the hg19.chrom.sizes file for the hg19 database is located at http://hgdownload.soe.ucsc.edu/goldenPath/hg19/bigZips/hg19.chrom.sizes. 4. Save the autoSql file interact.as to your computer. If you want the column labels to reflect non-directional interactions, you can change the default variable names from 'source...' and 'target...' to 'region1...' and 'region2...'. 5. Run bedToBigBed to create the bigInteract file: bedToBigBed -as=interact.as -type=bed5+13 interactExample4.inter.bed hg19.chrom.sizes interactExample4.inter.bb 6. Move the newly constructed bigInteract file to a web accessible http, https, or ftp location. 7. Construct a custom track line with a bigDataUrl parameter pointing to the newly created bigInteract file. track type=bigInteract name="interact Example Four" description="A bigInteract file" useScore=on bigDataUrl=/interactExample4.inter.bb visibility=pack browser position chr3:63,820,967-63,880,091 8. To fully take advantage of creating a bigInteract file, create a Track Hub and use a stanza such as the following: track exampleInteractTrack type bigInteract visibility full shortLabel exInteract longLabel Example interact track spectrum on scoreMin 175 maxHeightPixels 300:150:20 bigDataUrl interactExample4.inter.bb Understanding the interact file format This graphic represents the data in Example #2 with boxes around columns of data, separately illustrated as individual custom tracks in the lower image. [] The interact file format has 18 fields where the first 5 fields (box column1) are standard BED format fields, which define the span of the interaction to be viewed on a chromosome. In the below image, see the representation of box column1 and how it spans the length of each arc. Next, there are 3 fields for value, exp, and color before two sets of 5 fields that specify the coordinates, name, and strand of the source (box column2) and target (box column3) data, defining the endpoints of each interact arc. In the below image, the box column2 represents the left foot of each arc while the box column3 represents the right foot of each arc. The second row of the example data denotes an interaction to another chromosome, chr20, and thus is not represented by an arc. [] Sharing your data with others If you would like to share your interact/bigInteract data track with a colleague, learn how to create a URL by looking at Example 6 on this page. Extracting data from the bigInteract format Because bigInteract files are an extension of bigBed files, which are indexed binary files, it can be difficult to extract data from them. UCSC has developed the following programs to assist in working with bigBed formats, available from the binary utilities directory. - bigBedToBed — converts a bigBed file to ASCII BED format. - bigBedSummary — extracts summary information from a bigBed file. - bigBedInfo — prints out information about a bigBed file. Use the -as option to see the file field descriptions. As with all UCSC Genome Browser programs, one can type the program name (with no parameters) at the command line to view the usage statement. Troubleshooting If you encounter an error when you run the bedToBigBed program, check your input file for data coordinates that extend past the end of the chromosome. If these are present, run the bedClip program (available here) to remove the problematic row(s) in your input file before running the bedToBigBed program. 
/goldenPath/help/hgCodonColoringMrna.html:Genome_Browser_Codon_Coloring	Codon and Base Coloring for Alignment Tracks The Genome Browser's codon coloring feature allows users to quickly validate and compare alignment tracks, including mRNAs. To turn on codon coloring, select the desired option from the Color track by codons pull-down menu. Color Options - genomic codons - codons are labeled and colored according to the genomic sequence - mRNA codons - codons are labeled and colored according to the aligned sequence - nonsynonymous mRNA codons - codons are labeled and colored according to the aligned sequence. Only nonsynonymous codons are labeled. Conservative and non-conservative substitutions are defined using the BLOSUM62 score matrix. Conservative substitutions (yellow) are defined as positive BLOSUM scores, while non-conservative substitutions (red) are defined as negative scores. - mRNA bases - bases are labeled. - different mRNA bases - only those bases that differ from the genomic sequence are labeled and colored. Color Legend   Genomic/mRNA codons Nonsynonymous mRNA codons mRNA bases/different mRNA bases -------- ------------------------------------- -------------------------------------- --------------------------------- red "*" stop codons non-conservative codon substitutions base substitutions yellow   conservative codon substitutions   cyan spliced or truncated partial codons spliced or truncated partial codons   green start codons incl. methionine     black "X" truncated codons     Details The codon reading frame is defined by the GenBank CDS annotation of the aligned sequence. Each codon is colored and labeled in the direction of transcription. A sequence without a CDS will not be colored; these can be non-protein coding RNAs or an mRNA where a CDS annotation was not provided in the original data. The mRNA bases/different mRNA bases display options are colored and labeled in the direction of the genomic sequence. Note that it is possible to show the base complement, and thus change the base labeling, by clicking the arrow to the left of the base display or by clicking the "reverse" button. When zoomed out past the base level, the browser will choose one color to represent many bases. The priority of display, from most important to least important, is: different mRNA base/nonsynonymous codon coloring (if enabled), and then alignment coloring (if enabled). The browser will not display genomic/mRNA codon coloring when viewing large regions of the genome. To view labeling, the track must be zoomed to within 3 times the base level. For information about alignment insertion/deletion display options, click here. References Henikoff S and Henikoff JG. Amino acid substitution matrices from protein blocks. PNAS. 1992;89(22):10915-10919. 
/goldenPath/help/metadata.html:Adding_Track_Metadata	Adding metadata to tracks Contents Adding metadata to tracks Previous metadata versions Tagstorm metadata Tab-separated metadata Adding metadata to tracks Adding metadata to your tracks about cell lines, experimental protocols, or assays can be accomplished in a number of ways, via the newly supported metaDb or metaTab trackDb fields, or via the older style metadata trackDb field. The metaDb and metaTab fields link external tagStorm or tab-separated metadata files to the data in the hub. The new formats are preferred over the older metadata field, although the metadata lines will continue to be supported for track hubs, but no new features will be added as they will for tagStorm and tab-separated files. The following is an example of a genomes.txt file calling the tagStorm metadata file: genome hg38 metaDb relativePath/to/tagStorm.txt and specifying a tab-separated metadata file: genome hg38 metaTab relativePath/to/tabSep.txt When using tab-separated or tagStorm metadata, a meta column or line will be needed to specify which metadata information is applied to a track. The meta value should be a unique alphanumeric string. Previous metadata versions Currently, in order to add metadata to your tracks, you must specify all of the metadata key-value pairs in each stanza of a track that includes metadata, like the last line of the following example: track experiment1 shortLabel Donor A longLabel Donor A's Metadata Experiment type bigWig bigDataUrl http://genome.ucsc.edu/goldenPath/help/examples/bigWigExample.bw parent treatmentX on subGroups view=X metadata differentiation="10 hour" treatment=X donor=A lab="List Meta Lab" data_set_id=ucscTest1 access=group assay=long-RNA-seq enriched_in=exon life_stage=postpartum species="Homo sapiens" ucsc_db=hg38 Each track must have a separate metadata field and its own list of key-values, which can become cumbersome when each track in a group all share a common subset of metadata. For instance, if there are 10 tracks in a composite or multiWig, where each subtrack only differs in the "differentiation" tag, it would be more convenient to have a shared set of metadata and then specify the differences for each track. This is the motivation behind the tagStorm format, described below. You can find an example of a hub using the metadata example here and you can load the following session to view the hub, https://genome.ucsc.edu/s/PublicSessions/metadata_field. Tagstorm metadata The tagStorm format is a plain text file similar to a trackDb file that describes all of the tracks in a track hub, in that both are files where the first word in a line is the tag and the rest of the line is the value, and different stanza's are line delimited. TagStorm's are also similar to a spreadsheet, where a tag corresponds to a column and a stanza to an entire row. The tagStorm format is easy for computers to parse, reduce the redundancy of a tab-separated file, and they are human readable. Here is a canonical tagStorm example: lab tagStorm Lab data_set_id ucscTest1 access group assay long-RNA-seq enriched_in exon life_stage postpartum species Homo sapiens ucsc_db hg38 treatment X donor A differentiation 10 hour meta ucsc1_1 differentiation 1 day meta ucsc1_4 differentiation 5 days meta ucsc1_7 treatment Y donor B differentiation 10 hour meta ucsc1_2 differentiation 1 day meta ucsc1_5 differentiation 5 days meta ucsc1_8 donor C differentiation 10 hour meta ucsc1_3 differentiation 1 day meta ucsc1_6 differentiation 5 days meta ucsc1_9 Each stanza, such as "donor B", inherits from any stanzas above it at the right indentation level, and is a parent to stanzas beneath. In the example above, Treatment Y applies to both "donor B" and "donor C". Treatment X only applies to "donor A" as they are at the same indentation level. There are three differentiation times that apply to each of the donors and they can be referenced in the trackDb stanza using the meta line in the tagStorm file, i.e., meta ucsc1_1. The meta ucsc1_1 line would reference the following metadata: lab tagStorm Lab data_set_id ucscTest1 access group assay long-RNA-seq enriched_in exon life_stage postpartum species Homo sapiens ucsc_db hg38 treatment X donor A differentiation 10 hour A trackDb stanza using the tagStorm metadata can be seen in the following example: track experiment1 shortLabel Donor A longLabel Donor A's TagStorm Experiment type bigWig bigDataUrl http://genome.ucsc.edu/goldenPath/help/examples/bigWigExample.bw parent treatmentX on subGroups view=X meta ucsc1_1 You can find the complete example of the hub using the tagStorm metadata here and you can load the following session to view the hub, https://genome.ucssc.edu/s/PublicSessions/tagStorm_metadata. The hub uses a composite track, so if you are unfamiliar with composite tracks, the Quick Start Guide on composites can explain how the tracks are organized. The details page for the trackDb stanza example (Donor A) can be seen below. [] Tab-separated metadata Column or tab-separated metadata can be useful to store computer readable information as an array. While this format is very easy for a computer to parse, it can be bit confusing or difficult for humans to read and interpret. As you can see in the example below, many columns become redundant as they repeat the same information on each line. #lab data_set_id access assay enriched_in life_stage species ucsc_db treatment donor differentiation meta tabSepLab ucscTest1 group long-RNA-seq exon postpartum Homo sapiens hg38 X A 10 hour ucsc1_1 tabSepLab ucscTest1 group long-RNA-seq exon postpartum Homo sapiens hg38 X A 1 day ucsc1_4 tabSepLab ucscTest1 group long-RNA-seq exon postpartum Homo sapiens hg38 X A 5 days ucsc1_7 tabSepLab ucscTest1 group long-RNA-seq exon postpartum Homo sapiens hg38 Y B 10 hour ucsc1_2 tabSepLab ucscTest1 group long-RNA-seq exon postpartum Homo sapiens hg38 Y B 1 day ucsc1_5 tabSepLab ucscTest1 group long-RNA-seq exon postpartum Homo sapiens hg38 Y B 5 days ucsc1_8 tabSepLab ucscTest1 group long-RNA-seq exon postpartum Homo sapiens hg38 Y C 10 hour ucsc1_3 tabSepLab ucscTest1 group long-RNA-seq exon postpartum Homo sapiens hg38 Y C 1 day ucsc1_6 tabSepLab ucscTest1 group long-RNA-seq exon postpartum Homo sapiens hg38 Y C 5 days ucsc1_9 To reference a line in the TSV or CSV file in the trackDb stanza, a meta column must contain a unique alpha-numeric string. For example, ucsc1_1 would reference the following metadata in your track: #lab data_set_id access assay enriched_in life_stage species ucsc_db treatment donor differentiation meta tabSepLab ucscTest1 group long-RNA-seq exon postpartum Homo sapiens hg38 X A 10 hour ucsc1_1 A trackDb stanza using the tab-separated metadata can be seen in the following example: track experiment1 shortLabel Donor A longLabel Donor A's Tab Separated Experiment type bigWig bigDataUrl http://genome.ucsc.edu/goldenPath/help/examples/bigWigExample.bw parent treatmentX on subGroups view=X meta ucsc1_1 You can find the complete example of the hub using the tab-separated metadata here and you can load the following session to view the hub, https://genome.ucsc.edu/s/PublicSessions/TabSeparated_metadata. The hub uses a composite track, so if you are unfamiliar with composite tracks, the Quick Start Guide on composites can explain how the tracks are organized. The details page for the trackDb stanza example (Donor A) can be seen below. [] 
/goldenPath/help/hgVcfTrackHelp.html:Genome_Browser_VCF_Tracks	Configuring VCF tracks Genome Browser VCF tracks may be configured in a variety of ways to highlight different aspects of the displayed information. By default, VCFs will display alleles with base-specific coloring. Homozygote data are shown as one letter, while heterozygotes will be displayed with both letters. [VCF default display] The default VCF custom track will display colored bases and will not show clustering unless specified as VCF/tabix in the custom track page. The following section describes configuration settings available to VCF files compressed and indexed in the Tabix format. This requires VCF manipulation, separate index files, and a web accessible directory to reference from the bigDataUrl track line. For more information on setting up and uploading VCF/Tabix data, click the link on VCF custom track creation. Configuring the haplotype sorting display If the VCF file contains genotype columns for at least two samples (four haplotypes), then a haplotype sorting display can be configured. This can be useful for determining the similarity between the samples and inferring inheritance at a particular locus. Enable Haplotype sorting display: When this option is checked, each sample's phased and/or homozygous genotypes are split into haplotypes, clustered by similarity around a central variant, and sorted for display by their position in the clustering tree. The tree (as space allows) is drawn in the label area next to the track image. Leaf clusters, in which all haplotypes are identical (at least for the variants used in clustering), are colored purple. [VCF tree diagram] The haplotype tree can be seen to the left of the track. Each variant is drawn as a vertical column, using color to distinguish between reference alleles and alternate alleles of the horizontally running haplotypes. If unchecked, then the display is the same as for VCF without genotypes: a stacked bar graph of the top two alleles, showing the proportion of alleles if allele counts are available. This setting is enabled by default. The following options are applicable only when the haplotype sorting display is enabled: Haplotype sorting order: Haplotypes are sorted using a distance function that uses a central variant. Differences between haplotypes are penalized with weights that decrease for each successive variant away from the central variant. By default, the median variant in the window is used. By clicking on a variant in the display, you will get the option to always use that variant when it is in the current view. Haplotype coloring scheme: There are three ways that reference and alternate alleles can be colored: - By default, the reference allele is invisible and the alternate allele is black. When multiple haplotypes must be combined into the same pixel row, grayscale is used to shade according to the proportions of reference and alternate alleles. The central variant has a thin purple outline. Extra pixel rows at the top and bottom show the locations of variants in case they are hard to see due when the invisible reference allele is the major allele. Variants used in clustering have purple marks in these rows; variants outside the clustered regions have black marks. - The reference allele is blue and the alternate allele is red. Purple indicates a mix of reference and alternate alleles. The central variant has a thick black outline. - Both alleles are colored using the same color scheme as when there are no genotypes: A is red, C is blue, G is green and T is magenta. Gray indicates a mix of reference and alternate alleles. The central variant has a thick black outline. In all coloring modes, if some alleles in a haplotype are undefined, a pale yellowish color is used for those alleles. Haplotype clustering leaf shape: Leaf clusters are collections of identical haplotypes. By default, they are drawn as open triangles <. They can also be displayed as open rectangles [. [VCF options] Haplotype sorting display height: This number represents the track height in pixels. If the number of pixels is fewer than the number of haplotypes (2 * the number of genotype columns), some horizontal pixel rows must represent multiple haplotypes; with differing haplotypes' colors combined according to the selected coloring scheme. [VCF options] Filtering out variants Variants can be filtered out of the display according to several properties: - Exclude items with QUAL score less than N: If the checkbox is checked, then all variants whose QUAL column has a non-numeric value (e.g. ".") or a value less than N are excluded from display. By default, the checkbox is not checked and N is 0. - Exclude items with these FILTER values: This option appears only if the VCF header defines at least one FILTER code. There is a checkbox for each code defined in the header. If checked, then all variants with that code in the FILTER column are excluded from display. By default, no checkboxes are checked, so all variants are displayed regardless of FILTER column values. - Minimum minor allele frequency (if INFO column includes AF or AC+AN): If a variant's INFO field includes AF (alternate allele frequency) or both AC and AN (alternate allele count and total number of alleles), then its minor allele frequency can be compared against this threshold. If the minor allele frequency is less than the threshold, the variant will not be displayed. [VCF options] When you have finished making your configuration changes, click the Submit button to return to the annotation track display page. 
/goldenPath/help/chain.html:Genome_Browser_Chain_Format	Chain Format The chain format describes a pairwise alignment that allow gaps in both sequences simultaneously. Each set of chain alignments starts with a header line, contains one or more alignment data lines, and terminates with a blank line. The format is deliberately quite dense. Example: chain 4900 chrY 58368225 + 25985403 25985638 chr5 151006098 - 43257292 43257528 1 9 1 0 10 0 5 61 4 0 16 0 4 42 3 0 16 0 8 14 1 0 3 7 0 48 chain 4900 chrY 58368225 + 25985406 25985566 chr5 151006098 - 43549808 43549970 2 16 0 2 60 4 0 10 0 4 70 Header Lines chain score tName tSize tStrand tStart tEnd qName qSize qStrand qStart qEnd id The initial header line starts with the keyword chain, followed by 11 required attribute values, and ends with a blank line. The attributes include: - score -- chain score - tName -- chromosome (reference/target sequence) - tSize -- chromosome size (reference/target sequence) - tStrand -- strand (reference/target sequence) - tStart -- alignment start position (reference/target sequence) - tEnd -- alignment end position (reference/target sequence) - qName -- chromosome (query sequence) - qSize -- chromosome size (query sequence) - qStrand -- strand (query sequence) - qStart -- alignment start position (query sequence) - qEnd -- alignment end position (query sequence) - id -- chain ID The alignment start and end positions are represented as zero-based half-open intervals. For example, the first 100 bases of a sequence would be represented with start position = 0 and end position = 100, and the next 100 bases would be represented as start position = 100 and end position = 200. NOTE: When the strand value is "-",the query coordinates (qStart and qEnd) are on the reverse strand and must be subtracted from the chromosome size to obtain the correct position on the forward strand in the other genome. The reverse coordinates are subtracted as follows to get forward strand coordinates: qStartForward = qSize - qEnd qEndForward = qSize - qStart For example, using the query coordinates from chain 5 in hg38ToMm10.over.chain.gz: chain score tName tSize tStrand tStart tEnd qName qSize qStrand qStart qEnd id chain 442878230 chr1 248956422 + 158547112 207360161 chr1 195471971 - 21022354 65032227 5 The reverse strand coordinates are subtracted from the chromosome size: mm10Start = 195471971 - 65032227 = 130439744 mm10End = 195471971 - 21022354 = 174449617 The forward strand coordinates for chain 5 on mm10 are chr1 130439744 174449617, or with 1-based coordinates for a position range, chr1:130,439,745-174,449,617. To reverse the calculation and derive the corresponding hg38 coordinates using chain 5 in mm10ToHg38.over.chain.gz, note that the derived mm10 coordinates match the tStart and tEnd values: chain 442878230 chr1 195471971 + 130439744 174449617 chr1 248956422 - 41596261 90409310 5 The hg38 coordinates are subtracted as follows: hg38Start = 248956422 - 90409310 = 158547112 hg38End = 248956422 - 41596261 = 207360161 These coordinates match the target coordinates in hg38ToMm10.over.chain.gz. Alignment Data Lines Alignment data lines contain three required attribute values: size dt dq - size -- the size of the ungapped alignment - dt -- the difference between the end of this block and the beginning of the next block (reference/target sequence) - dq -- the difference between the end of this block and the beginning of the next block (query sequence) NOTE: The last line of the alignment section contains only one number: the ungapped alignment size of the last block. "Snake" rearrangement display Rearrangement display, sometimes called snakes display, is an alternative way to view pairwise alignments. It is available for PSL and chain format tracks. Rearrangement display is a representation of the path that the sequence follows in the "other" sequence. You start in the upper left and move to the right, following the lines if you come to the end of a block. If a block is red, which means it is a match on the negative strand, then you reverse your course and start going from right to left. The gray lines mean there are no bases in the other sequence between the blocks. Orange lines means there are some bases in there that are not aligning. The display type can be enabled on the track configuration page of eligible tracks. Below are two examples for clarity. [Rearrangement display example 1] This example shows a tandem duplication on CDH1 which duplicates 3 exons. [Rearrangement display example 2] This example shows an inversion, which can be identified by the red colored sequence. 
/goldenPath/help/docker.html:Docker_help_page	Docker Help Page Contents What is Docker? How to Install Docker Desktop? Using Docker Desktop for UCSC Genome Browser Create a Docker Volume for Data Persistence Updating the Latest UCSC Genome Browser Version Customize a UCSC Genome Browser Docker Container What is Docker? Docker is a platform for developing, testing and running applications. Docker can be used to run genomics tools and manage software such as the UCSC Genome Browser. Docker offers consistency across different computers and environments by packaging everything needed including specific software versions and configurations into a self-contained unit called a container. Container A container is software that packages up code and all its dependencies to run an application quickly and reliably from one computing environment to another. A container is isolated from other containers and a Docker container can be run on a developer's local laptop, virtual machines, on cloud providers, or other combinations of environments. A genomics analysis pipeline or entire analysis environment can be packaged to a local computer into a Docker container and moved to a cluster or a cloud server. See also: Use containers to Build, Share and Run applications Image A Dockerfile is a text file that provides instructions to build an image. The Dockerfile is written in Dockerfile syntax. A docker image is a read-only template with instructions and everything needed to run an application for the container. See also: Overview of the get started guide How to Install Docker Desktop? Windows - Go to the Install Docker Desktop on Windows page - Check system requirements - Install Docker interactively or from the command line macOS - Go to the Install and run Docker Desktop on Mac page - Check system requirements - Install Docker interactively or from the command line Linux - Go to the Install Docker Desktop on Linux page and select Linux distribution - Check system requirements - Follow Generic installation steps Using Docker Desktop for UCSC Genome Browser Start Docker Desktop after installation is complete: - Windows: start Docker Desktop from the Start menu - macOS: start Docker Desktop from the Applications folder - Linux: start the Docker service by running the following command on the terminal: sudo systemctl start docker Obtaining a UCSC Genome Browser Dockerfile The UCSC Genome Browser dockerfile can be obtained from the UCSC Genome Browser Github by using the wget command: wget https://raw.githubusercontent.com/ucscGenomeBrowser/kent/master/src/product/installer/docker/Dockerfile Creating a Image Once the dockerfile has been downloaded, running the docker build with the 't' option allows the naming and the optional tag (format: "name:tag") of the image. The image can be created by running the following command in the same directory where the dockerfile is located: docker build . -t user_name/ucsc_genomebrowser_image Creating a Container After the image has been created, running the docker run command and the image with the -d option allows the container to be run in the background, whereas the default runs the container in the foreground. The -p option publishes a container's port(s) to the host. The following command maps port 8080 on the host machine to port 80 in the container and names the container using the -name option: docker run -d --name ucsc_genomebrowser_container -p 8080:80 user_name/ucsc_genomebrowser_image Accessing the running container via http://localhost:8080 Running the following command will list the running container: docker container ls Running the following command stops the running container:: docker stop <container_name_or_id> Running the following command removes the existing container:: docker rm <container_name_or_id> Using Docker Desktop to Create a Container The Docker Desktop user interface can be used to run the container by going to the images tab and clicking the run button under Actions: [] Click Optional settings in the "Run a new container" pop-up window: [] Enter a Container name and a Host port in the "Run a new container" popup window: [] Click the link with the Host port to go to the running container via localhost: [] Create a Docker Volume for Data Persistence A Docker volume allows the data to be persistent (long-lasting) after the container restarts and mount to a host directory or another container's data volume into the UCSC Genome Browser container. The following command creates a new volume named ucsc_genomebrowser_volume that containers can consume and store data in: docker volume create ucsc_genomebrowser_volume After creating the volume named ucsc_genomebrowser_volume, running the docker run command starts the UCSC Genome Browser container using the user_name/ucsc_genomebrowser_image image and the -v option to mount the volume created in the previous step. docker run -d --name ucsc_genomebrowser_volume -p 8080:80 -v ucsc_genomebrowser_volume:/data user_name/ucsc_genomebrowser_image Files can be copied into the Docker volume or a bind mount can be used to link a host directory containing data to the /data directory inside the container. The following command copies a file to the data directory inside the container: docker cp file.txt ucsc_genomebrowser_volume:/data Running the execute command will list the file inside the running container: docker exec ucsc_genomebrowser_volume ls data Updating the Latest UCSC Genome Browser Software Access the Docker Container's Shell Updating the latest UCSC Genome Browser version will require access to the Docker container running shell (command-line interface) of the UCSC Genome Browser. The execute command can be run inside a running Docker container with the -it options. The -i or --interactive option allows interaction with the command being executed and keeps STDIN open even if not attached. This will allow input to be provided for the command. The -t or --tty option allocates a pseudo-TTY and allows for a more interactive experience. The following example shows how to run exec command and the -it options: docker exec -it <container_name_or_id> /bin/bash Update the Genome Browser Software Running the following command updates the Genome Browser software: bash root/browserSetup.sh cgiUpdate Customize a UCSC Genome Browser Docker Container Editing hg.conf The hg.conf file is a file that has information on how to connect to MariaDB, the location of the other directories and various other settings. The hg.conf file can be edited by running the execute command inside a running Docker container with the -it options. The -i or --interactive option allows interaction with the command being executed and keeps STDIN open even if not attached. This will allow input to be provided for the command. The -t or --tty option allocates a pseudo-TTY and allows for a more interactive experience. Any common text editors such as vi, nano, and vim can be used with the execute command and the -it options. The following example shows how to edit the hg.conf file using vi: docker exec -it <container_name_or_id> vi /usr/local/apache/cgi-bin/hg.conf Changing the Default Genome Browser Options Track settings such as fonts, text size, default tracks, attached hubs, and the default region can be customized and set as the default settings. These settings will appear every time the UCSC Genome Browser graphic display is opened and will also appear after a reset of all user settings. This can be useful when working with a different assembly than hg38, having track hubs automatically attached, or changing the visibility of tracks. - The first step is to create a Session containing all desired display options, hubs, tracks, and default region in the UCSC Genome Browser Docker container. Open the UCSC Genome Browser Docker container shell. See the Access the Docker Container's Shell section of this page. - Create a new file, 'defaultCart.sql', to make a new MySQL table. Add the following to the defaultCart.sql file: #The default cart CREATE TABLE defaultCart ( contents longblob not null # cart contents ); - Drop the existing defaultCart table by running the following query: mysql hgcentral -Ne "DROP TABLE defaultCart" - Load the defaultCart.sql file as a table by running the following query: mysql hgcentral < defaultCart.sql - Insert the session to the default cart table by using the user name and the session name, which was the session saved in the earlier step, and run the following query (add userName): mysql hgcentral -Ne "insert into defaultCart select contents from namedSessionDb where sessionName='nameOfSession' and userName='nameOfUser'" 
/goldenPath/help/hubQuickStartAssembly.html:Assembly_Hub_Quick_Start	Quick Start Guide to Assembly Hubs Assembly Hubs allow researchers to create Track Data Hubs on assemblies that are not in the UCSC Browser. By including the underlying reference sequence in UCSC twoBit format, as well as data tracks, researchers can browse and annotate any genome. We may have a GenArk Hub of your genome, or you can visit our assembly request page and we can build an assembly hub for you. For more information please refer to the Assembly Hub Wiki. Below is also a section about starting GBiB Assembly Hubs. STEP 1: In a publicly accessible directory, copy this Arabidopsis thaliana plant assembly hub, which includes an araTha1.2bit file, using the following wget command: wget -r --no-parent --reject "index.html*" -nH --cut-dirs=3 http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubAssembly/plantAraTha1/ Alternatively, if you do not have wget installed, you can curl these files individually. Perform the curl -O option in the location you wish to copy the files: curl -O http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubAssembly/plantAraTha1/hub.txt If you use curl, be sure to recreate the structure with matching araTha1 and araTha1/bbi directories. Double check you have all the files by looking here: http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubAssembly/plantAraTha1/ STEP 2: Paste your hub.txt link (http://yourURL/hub.txt) into the Connected Hubs tab of the Track Data Hubs page, click the "Add Hub" button, and then click the "Genome Browser" link from the top bar. Alternatively build a URL that will directly load your assembly hub and display it on hgGateway. Then click the "Genome Browser" link from the top bar to view your assembly hub: http://genome.ucsc.edu/cgi-bin/hgHubConnect?hgHub_do_redirect=on&hgHubConnect.remakeTrackHub=on&hgHub_do_firstDb=1&hubUrl=http://yourURL/hub.txt This URL should work the same as using the original data just copied: http://genome.ucsc.edu/cgi-bin/hgHubConnect?hgHub_do_redirect=on&hgHubConnect.remakeTrackHub=on&hgHub_do_firstDb=1&hubUrl=http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubAssembly/plantAraTha1/hub.txt STEP 3: Congratulations! Your assembly hub should display! If you are having problems, be sure all your files and the directories are publicly accessible. You may also wish to reset the browser occasionally to clear all existing data. For hubs to work, your server must also accept byte-ranges. You can check using the following command to verify "Accept-Ranges: bytes" displays: curl -I http://yourURL/hub.txt Now that you have the assembly hub copied from above, you can copy the directory and start to edit some of the documents such as genomes.txt, groups.txt, and trackDb.txt to understand how they work. Refer to the Assembly Hub Wiki to understand how to build a twoBit file for your own original fasta files. Read more about trackDb settings in the definition document. This assembly hub is a an abbreviated version of a larger plant assembly Public Hub. You can explore the larger hub structure here. Please note that the Browser waits 5 minutes before checking for any changes to these files. When editing hub.txt, genomes.txt, trackDb.txt, and related hub files, shorten this delay by adding udcTimeout=1 to your URL. For more information, please see the Debugging and Updating Track Hubs section of the Track Hub User Guide. Also, for more detailed instructions on setting up a regular hub, please see the Setting Up Your Own Track Hub section of the Track Hub User Guide. Setting up Blat and In-Silico PCR for an Assembly Hub By running gfServers from your institution, you can enable blat on your assembly hubs. See Starting Blat and In-Silico PCR for an Assembly Hub for details. Setting up an Assembly Hub on GBiB with Blat and In-Silico PCR included With an operational installation of Genome Browser in a Box (GBiB), you can quickly and easily acquire an example assembly hub and run gfServers locally on the GBiB to enable Blat and In-Silico PCR. See the section Starting a Blat and In-Silico PCR enabled Assembly Hub on GBiB for more information. Resources - Assembly Hubs Wiki - Track Hub User Guide - Track Database (trackDb) Definition Document - Public Hub Guidelines - Quick Start Guide to a Basic Hub - Quick Start Guide to Organizing Hubs Starting Blat and In-Silico PCR for an Assembly Hub From the location of yourAssembly.2bit file, http://yourURL/yourAssembly/yourAssembly.2bit, you can start two gfServers, specifying a port for the assembly hub to access amino acid sequence, 17777 -trans, or DNA sequence, 17779, in this example: gfServer start localhost 17777 -trans -mask yourAssembly.2bit & gfServer start localhost 17779 -stepSize=5 yourAssembly.2bit & Then you can edit the genomes.txt file of your assembly hub to include three lines in the stanza referring to yourAssembly, that would have matching port numbers: transBlat yourLab.yourInstitution.edu 17777 blat yourLab.yourInstitution.edu 17779 isPcr yourLab.yourInstitution.edu 17779 The assembly hub can be configured to talk to a dynamic BLAT server that loads a pre-built index when started by an xinetd super-server. This allows genomes to have a blat server without needing it to be resident in memory at all times. See Running your own gfServer and Adding BLAT servers for details on how to setup dynamic BLAT servers See an example genomes.txt with commented out lines here, and please note the uppercase "B" in transBlat. For more information, see the "Adding BLAT servers" section of the Assembly Hub Wiki. The Source Downloads page offers access to utilities with pre-compiled binaries such as gfServer found in a blat/ directory for your machine type here and further blat documentation here. Please note that because the -mask option in the above 17777 -trans gfServer option will mask all lower-case sequence from being matched, you may not wish to include it. See the above blat links and gfServer usage statement for more information. If you have trouble connecting your blat servers with the browser or if the browser cannot access your files, check if your institution has a firewall that prevents the browser from sending multiple inquiries. If this is the case, ask your systems administrator to add the following IP addresses as exceptions so that access is not limited. 128.114.119.* 129.70.40.99 134.160.84.67 128.114.198.32 This will allow connections with the U.S.-based genome.ucsc.edu site, the Europe-based mirror, the Asia-based mirror, and the UCSC development server. Starting a Blat and In-Silico PCR enabled Assembly Hub on GBiB STEP 1. Acquire and install Genome Browser in a Box: http://genome.ucsc.edu/goldenPath/help/gbib.html. You may also wish to read this UCSC blog post. STEP 2. With your GBiB operational, use your computer's terminal program to ssh into your GBiB: ssh browser@localhost -p 1235, using browser for the password. STEP 3. Navigate to the GBiB's /folders directory and use sudo to wget this assembly hub: cd /folders sudo wget -r --no-parent --reject "index.html*" -nH --cut-dirs=3 http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubAssembly/plantAraTha1/ STEP 4. You now have all the required files on your local machine and can load this plant assembly hub by using this URL and selecting it under the "group" category where "Plant araTha1" displays: http://127.0.0.1:1234/cgi-bin/hgGateway?genome=araTha1&hubUrl=http://127.0.0.1:1234/folders/hubExamples/hubAssembly/plantAraTha1/hub.txt STEP 5. To enable blat you must acquire the gfServer utility. The UCSC Genome Browser and Blat software are free for academic, nonprofit, and personal use. Commercial download and installation of the Blat and In-Silico PCR software may be licensed through Kent Informatics. You can obtain just the gfServer utility on your GBiB with either of the following commands that will create a bin directory and install the tool. The commands use the North American and the European download servers respectively. mkdir ~/bin -p; rsync -avP hgdownload.soe.ucsc.edu::genome/admin/exe/linux.x86_64/blat/gfServer ~/bin/ mkdir ~/bin -p; rsync -avP hgdownload-euro.soe.ucsc.edu::genome/admin/exe/linux.x86_64/blat/gfServer ~/bin/ The GBiB also includes a tool you can run on the command line to download an entire suite of tools including gfServer: gbibAddTools STEP 6. Navigate to the genomes.txt file of this assembly hub: cd /folders/hubExamples/hubAssembly/plantAraTha1/ Edit the currently commented-out blat lines with sudo vi genomes.txt and use "x" when the cursor is over the # at the start of the line to remove it and :w! to save the changes and :q to quit. blat localhost 17779 transBlat localhost 17777 isPcr yourLab.yourInstitution.edu 17779 Please note that if you loaded your hub earlier, it will take five minutes (300 seconds) for the browser to check for any changes to genomes.txt, and that this delay can be shortened temporarily by adding &udcTimeout=10 to the URL. See more information in the Debugging and Updating section of the Track Hub User Guide. STEP 7. Change directories to the 2bit file: cd /folders/hubExamples/hubAssembly/plantAraTha1/araTha1 Run the two gfServer commands to start the blat servers: gfServer start localhost 17777 -trans -mask araTha1.2bit & gfServer start localhost 17779 -stepSize=5 araTha1.2bit & STEP 8. Load this plant assembly hub by using this URL and selecting it under the "group" category where "Plant araTha1" displays: http://127.0.0.1:1234/cgi-bin/hgGateway?genome=araTha1&hubUrl=http://127.0.0.1:1234/folders/hubExamples/hubAssembly/plantAraTha1/hub.txt On the blat page, http://127.0.0.1:1234/cgi-bin/hgBlat, you can now select the Arabidopsis thaliana assembly and blat plant amino acid sequences, such as IYQTRENKYIIGEIQITESERDRRRSSLPGNH or DNA sequences, such as TAAGTAAAAAATAATATGATTAAGACTAATAAATCTTAATAGTTAATACT. On the PCR page, http://127.0.0.1:1234/cgi-bin/hgPcr, you can now select the Arabidopsis thaliana genome and enter a forward primer such as TAGGTCTGCACCTGTGGTTCAAAATTTT and a reverse primer such as CAATACAAGTCAACATTTTAGCGCCGAGA and click the "Flip Reverse Primer" box and then click submit to find matches on the assembly. 
/goldenPath/help/qValue.html:Genome_Browser_Q-Value	Q-Value For any genome-wide analysis, reporting individual p-values can be misleading, because the p-value does not correct for the large number of tests performed. The q-value is an analog of the p-value that incorporates multiple testing correction. The q-value is defined as the minimum false discovery rate at which an observed score is deemed significant. Thus, the q-value attempts to control the percentage of false positives among a collection of scores. This contrasts with a traditional Bonferroni correction (or E-value), which controls the probability of one or more false positives in a collection of scores. Software for computing q-values from a collection of p-values is available at: https://github.com/StoreyLab/qvalue For a good introduction to false discovery rate estimation and the q-value see: Storey JD, Tibshirani R. Statistical significance for genomewide studies. Proc Natl Acad Sci. 2003 Aug 5;100(16):9440-5. 
/goldenPath/help/decorator.html:Genome_Browser_Decorators	Track Decorators Overview Track decorators allow highlighting parts of features with colors and/or symbols (glyphs/shapes) within a single track. The decorations can either be overlaid onto the feature or shown directly underneath. Decorators can be added to BED 12+, bigBed, PSL, and bigGenePred tracks. To add decorations to a track, the decorations must first be stored in a separate bigBed file that includes extra fields to identify the decorated items. The track settings for the track must then be modified to include a pointer to that bigBed file; this can be done either in the track line for a custom track, or in the trackDb.txt file for a hub. At present, we only support a single decorator per track, but we anticipate supporting multiple decorators in the future. [] Contents Annotating the Genome The Decorator Styles Getting Started Examples Hub TrackDb Settings Troubleshooting Annotating the Genome The genome browser‘s primary way to annotate the genome uses colored rectangles ("exons" for gene tracks) linked by thin lines ("introns"), often stored as a bigBed. These were originally used for genes but then evolved to cover other types of annotations, e.g. enhancers, chromatin modifications, or single nucleotide variants. We usually call these annotations "features". Each rectangle ("exon") of a feature has the same color and individual parts cannot be highlighted. If you wanted to highlight parts of the features, traditionally this required a second track. [] A track decorator is more compact than creating a separate track for these sub-features, but loses some of the abilities of normal tracks, for example, there is no right-click on the sub-features and the user cannot click these sub-features. The primary use case driving this display was protein-domains drawn on top of gene models, but we have found many other applications since then, e.g. drawing summits on ATAC peaks or highlighting selenocysteines on transcripts. The Decorator Styles Track decorators can be shown in two styles, "block" and "glyph" style. The "block" style option can be used to color exons and introns and can display a label for them. For example, the "block" track decorator could be used to overlay protein domain boundaries on transcripts where usually one would use an entirely different track for the domains. [] The "glyph" style option offers 8 different types of glyphs and the color of choice. [] The "glyph" style option can be used to draw entirely new symbols, for example, to indicate insertion positions on the genome with small triangles. We appreciate user feedback. If you have glyph questions, glyph style requests, or have found new glyph applications, please contact our mailing list. To use decorators in your track hub or custom tracks, you will need to create an additional bigBed file that defines the regions, colors, and glyphs. Getting Started The Decorator bigBed A decorator bigBed file, which contains decorations for annotating another track, is very similar to our standard bigBed file format. The only difference is the addition of some extra required fields, which describe how each decoration should be drawn and what item within that other track it annotates. The full .as format for decorator bigBed files is as follows: string chrom; "Chromosome (or contig, scaffold, etc.)" uint chromStart; "Start position in chromosome" uint chromEnd; "End position in chromosome" string name; "Name of item" uint score; "Score from 0-1000" char[1] strand; "+ or -" uint thickStart; "Start of where display should be thick (start codon)" uint thickEnd; "End of where display should be thick (stop codon)" uint color; "Primary RGB color for the decoration" int blockCount; "Number of blocks" int[blockCount] blockSizes; "Comma separated list of block sizes" int[blockCount] chromStarts; "Start positions relative to chromStart" string decoratedItem; "Identity of the decorated item in chr:start-end:item_name format" string style; "Draw style for the decoration (e.g. block, glyph)" string fillColor; "Secondary color to use for filling decoration, blocks, supports RGBA" string glyph; "The glyph to draw in glyph mode; ignored for other styles" A copy of this file can be found here. Valid values for the style field are "block" and "glyph". Valid glyph entries include "Circle", "Square", "Diamond", "Triangle", "InvTriangle", "Octagon", "Star", and "Pentagram". If the text isn't recognized, Circle will be used by default. The "decoratedItem" field (chr:start-end:item_name format) captures the link between the decoration and what item in the track is being decorated. The contents of this field must be the chromosome, BED start coordinate, BED end coordinate, and item name for the decorated item (note - these are 0-based half-open BED coordinates, not 1-based fully closed coordinates. That means they are the same values as should appear in a BED file describing the decorated item). Examples Example #1: Building a decorator bigBed Here is an example of how to use this format. Consider the item in BED format as a "feature": chr1 1000 2000 feature 0 + 1000 2000 0 2 400,400 0,600 We can take this BED file and construct a mainEx.bb from it as described in the bigBed documentation. To add a decoration to the "feature" item that highlights the region at position chr1:1,201-1,800, we could create a corresponding item in a decorator bed file like the following: chr1 1200 1800 highlight 0 + 1200 1800 255,0,0,255 1 600 0 chr1:1000-2000:feature block 255,0,0,128 Ignored - The name of this decorator item is highlight specified in the fourth field of the decorator bed file. - The 9th field of the decorator bed file (255,0,0,255) specifies the decoration outline using RGB values and an alpha value to control opacity. - The chr1:1000-2000:feature (13th field of the decorator bed file) entry describes which item in the main bed file is to be annotated. In this case, it's an item with the name "feature" at position chr1:1,001-2,000. - The 15th field of the decorator bed file (255,0,0,128) specifies the interior of the decoration using RGB values and an alpha value to control opacity. - The Ignored value is used in the last field of the bed file because we are creating a block decoration (a decoration that annotates a range of bases). To add a glyph decoration that marks the final base of the transcript with a green circle, we would then include the following line in the decorator bed file: chr1 1999 2000 green_circle 0 + 1999 2000 0,255,0,255 1 1 0 chr1:1000-2000:feature glyph 0,255,0,255 Circle - The name of this decorator item is green_circle specified in the fourth field of the decorator bed file. - The 9th field of the decorator bed file (0,255,0,255) specifies the decoration outline using RGB values and an alpha value to control opacity. - The chr1:1000-2000:feature (13th field of the decorator bed file) entry describes which item in the main bed file is to be annotated. - The 15th field of the decorator bed file (0,255,0,255) specifies the interior of the decoration using RGB values and an alpha value to control opacity. - The Circle glyph style is specified in the last field of the decorator bed file. Both of these bed decorations can be stored in a file named decoratorsEx.bed and then built as a decoratorsEx.bb using the hg38.chrom.sizes and the decoration.as files and running the following command: bedToBigBed -type=bed12+ -as=decoration.as decoratorsEx.bed hg38.chrom.sizes decoratorsEx.bb Debugging the decorator bigBed The resulting decoratorsEx.bb file can be displayed as a stand-alone custom track for debugging purposes. This process allows verification that the decorations are correctly applied. To display the decorator bigBed, navigate to the UCSC Genome Browser Custom Tracks page and paste the URL into the designated text field: browser position chr1:1000-2000 https://genome.ucsc.edu/goldenPath/help/examples/decorator/decoratorsEx.bb Once the decorator bigBed is loaded, the decorations will be rendered on the UCSC Genome Browser, allowing for verification of the correct display. Example #2: Create a custom track Create a decorator bigBed custom track using the decorator bigBed file from Example #1. 1. Construct a track line that references the bigBed and the decorator bigBed file: browser position chr1:1000-2000 track type=bigBed name="Decorators Example Two" description="bigBed with decorators" visibility=pack bigDataUrl=https://genome.ucsc.edu/goldenPath/help/examples/decorator/mainEx.bb decorator.default.bigDataUrl=https://genome.ucsc.edu/goldenPath/help/examples/decorator/decoratorsEx.bb 2. Paste the track line into the custom track page for the human assembly, hg38. 3. Click the Submit button. Custom tracks can also be loaded via one URL line. The link below loads the same bigBed with decorators track and sets additional parameters in the URL: https://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&hgct_customText=track%20type=bigBed%20bigDataUrl=https://genome.ucsc.edu/goldenPath/help/examples/decorator/mainEx.bb%20decorator.default.bigDataUrl=https://genome.ucsc.edu/goldenPath/help/examples/decorator/decoratorsEx.bb&position=chr1:1000-2000 Example #3: Create a decorator bigBed with extra (custom) fields Additional fields can also be added onto the end of the .as file, though they will not be used by default. The additional fields can be used for custom feature filters and mouseOvers options. For example, to set up a filterValues filter, you would add the following two fields to the decoration.as file: int numKeywords; "Number of keywords" string[numKeywords] keywords; "Keywords for this decorator" The numKeywords field specifies the number of keywords and the keywords field specifies the keywords. You would then modify your decoratorsEx.bed file to include an additional fields at the end of each line, detailing which keywords apply to each of the decorations: chr1 1200 1800 highlight 0 + 1200 1800 255,0,0,255 1 600 0 chr1:1000-2000:feature block 255,0,0,128 Ignored 1 Type1 chr1 1999 2000 green_circle 0 + 1999 2000 0,255,0,255 1 1 0 chr1:1000-2000:feature glyph 0,255,0,255 Circle 2 Type2,Type3 The first bed entry filters for one class "Type1" and the second bed entry filters for two classes "Type2,Type3". You can also add a mouseOverField which will allow you to mouse over text that is different from the "name" of the decorator bigBed. You would add the following field to the decoration.as file: string mouseOverField; "Mouse over text" Then modify your decoratorsEx.bed file to include the additional field. chr1 1200 1800 highlight 0 + 1200 1800 255,0,0,255 1 600 0 chr1:1000-2000:feature block 255,0,0,128 Ignored 1 Type1 alternate_highlight_name chr1 1999 2000 green_circle 0 + 1999 2000 0,255,0,255 1 1 0 chr1:1000-2000:feature glyph 0,255,0,255 Circle 2 Type2,Type3 alternate_green_circle_name You would then rebuild the decorations.bb file using the decorationEx_fields.as with extra fields: bedToBigBed -type=bed12+ -as=decorationEx_fields.as decoratorsEx.bed hg38.chrom.sizes decoratorsEx.bb Hub TrackDb Settings The statement below provides trackDb settings to add decorators to a track hub: track testTrack type bigBed 12 bigDataUrl https://genome.ucsc.edu/goldenPath/help/examples/decorator/mainEx.bb itemRgb on decorator.default.bigDataUrl https://genome.ucsc.edu/goldenPath/help/examples/decorator/decorationsEx.bb - The type can be BED 12+, bigBed, PSL, and bigGenePred tracks. - The bigDataUrl is the main file to be annotated for decorators. - The itemRgb setting allows the coloring of the interior of the block decorators. - The decorator.default.bigDataUrl setting adds decorations to the track and will point to the bigBed file containing the decorators. Other settings are available to further configure decorators. Each may be applied to the decorator instead of the primary track by prepending "decorator.default." to the setting. For example, to set up a filterValues filter on the "keywords" field of the decorator, allowing the user to filter to any combination of three classes "Type1", "Type2", and "Type3", the following trackDb entry could be used were the multipleListOr setting splits the three classes list values by commas: decorator.default.filterValues.keywords Type1,Type2,Type3 decorator.default.filterType.keywords multipleListOr Please note that this would also require building an extra keywords field into the decorator bigBed to hold those values, see Example #3 for more details. Decorators also support the mouseOver and mouseOverField settings that can be applied to bigBed tracks: decorator.default.mouseOver decorator $name mouseOver decorator.default.mouseOver $mouseOver You can configure the block decoration placement visibility to "overlay", "adjacent", or "hide" using the blockMode setting: decorator.default.blockMode adjacent You can use the maxLabelBases setting to set a maximum window size (in bases) for which labels will be drawn. If not set, the value will default to 200kb. This can be useful to deactivate decoration labels when there are too many track items and too many decoration labels to process visually. decorator.default.maxLabelBases 100000 A full list of supported decorator settings is available in the trackDb documentation. Troubleshooting If you get an error when you run the bedToBigBed program, please check your input BED file for data coordinates that extend past the end of the chromosome. If these are present, run the bedClip program (available here) to remove the problematic row(s) in your input BED file before using the bedToBigBed program. 
/goldenPath/help/customColumn.html:Genome_Sorter_Custom_Columns	Displaying Your Own Columns in the UCSC Gene Sorter The Gene Sorter provides dozens of columns containing information on genes computed at UCSC or provided by outside collaborators. In addition to these standard columns, users may also upload their own columns for temporary display in the browser. Custom columns are viewable only on the machine from which they were uploaded and are kept only for 8 hours after the last time they were accessed. Optionally, users can make custom columns viewable by others as well. Gene Sorter custom columns are based on files in line-oriented format. Each column is described by an initial column line followed by one or more data lines. The column line describes the name, hyperlinks, and other overall characteristics of the column. Each data line contains specific information about a gene annotated by the column. Lines starting with # are ignored. Only one column file may be loaded at a time; however, multiple column descriptions may be included in the same custom file, separated by blank lines. The column line Each column description must begin with a column line containing the keyword column followed by an optional set of one or more attribute pairs: column [attribute1]=[value1] [attribute2]=[value2]... Attribute values must be enclosed in quotes if they contain spaces or tabs. Attribute names and data values are case-sensitive. The following attributes may be defined: name - Symbolic name of the custom column (not displayed to user). shortLabel - Label displayed at the top of the column in the Gene Sorter display. The default value is User Column. longLabel - Short description of the column displayed after the name on the configuration and filter pages. The default value is User custom column. visibility - Controls whether column is displayed by default: on = display column, off = hide column. The default is on. priority - Specifies the display order of the column relative to others. Columns with lower priority values appear toward the lefthand side of the display. The standard browser columns have priorities between 0 and 20. The default priority is 2.01. itemUrl - URL used to construct hyperlinks accessed by clicking on column data values. If the URL contains %s, the column value will be inserted at that position in the hyperlink string. For example, if itemUrl for a column is defined as http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg16& position=%s (the UCSC Genome Browser URL), clicking on the data value NM_024014 will open the Human Jul. 2003 Genome Browser to the position occupied by RefSeq accession NM_024014. There is no default for this attribute. labelUrl - URL of the hyperlink accessed by clicking on the column label. No default. search - When the attribute is set to one of the following values, column data may be searched using the Gene Sorter search text box. Rows containing matches will be moved to the top of the display. By default, no search criterion is set. - exact - matches only if the text entered in the position search box exactly matches the column data text. - prefix - matches if search text exactly matches the initial part of the column data text. - fuzzy - matches if search text matches any portion of the column data text. idLookup - When set, this attribute specifies the standard column that should be used to link key values to the Gene Sorter display. For example, if idLookup is set to refSeq, a custom column data row containing the key NM_024014 will display on the same row as the RefSeq row containing NM_024014. The idLookup values are case-sensitive. By default, idLookup is set to the acc (GenBank) column. To determine the idLookup value that corresponds to a specific standard column, click the column's title in the Gene Sorter display (use the configure button to turn on the column display if it is currently hidden). The near.do.colInfo parameter in the URL linked to the column title is set to the idLookup value that corresponds to that column. isNumber - When this attribute is set to on, the filter page displays numerical max/min controls for this column. Default is off. Data lines Data lines are of the format: [key] [value] - key - links the custom column data value to a data value in the column specified by the idLookup attribute. If idLookup is unset, the browser looks for a match in the acc (GenBank) column. - value - data value to be displayed in the custom column row that matches the specified key. It is permissible to have more than one key/value pair per key. In this case, the column displays a comma-separated list of values. Data line keys and values are case-sensitive. Examples Example #1 This example defines a custom column for the Jul. 2003 Gene Sorter. The column's key values are linked to data in the refSeq column. Column rows can be filtered by numerical range by setting the max/min values for the column on the filter page. #Custom column file for MyLab Trial 3 # #Column line: # column name="MyLab Data" shortLabel="MyLab" longLabel="MyLab Trial 3" visibility=on priority=2.05 idLookup=refSeq isNumber=on # #Data lines (key links to refSeq column): # NM_005523 1.2 NM_005522 4.5 NM_018951 5.1 NM_000522 5.7 NM_030661 9.4 NM_002141 5.2 NM_024014 4.3 NM_006896 6.0 Example #2 This example defines a custom column for the Oct. 2003 mouse Gene Sorter. The column's key values are linked by default to the acc (GenBank) column. Clicking on the column's title (UCSCLab) displays the web page http://genome.ucsc.edu/. Clicking on a specific data value displays the web page http://genome.ucsc.edu/cgi-bin/hgTracks?db=mm4 (the UCSC Genome Browser) at the position specified by the data value. A search on the word MOUSE will display a list of all UCSC BioLab data that contains the string "MOUSE". #Custom column file for UCSC BioLab Test Data # #Column line: # column name="UCSCLab Data" shortLabel="UCSCLab" longLabel="UCSC BioLab Test Data 4/4/04" visibility=on priority=2.05 itemUrl=http://genome.ucsc.edu/cgi-bin/hgTracks?db=mm4&position=%s labelUrl=http://genome.ucsc.edu search=fuzzy # #Data lines (key links to refSeq column): # U20370 HXAB_MOUSE L08757 HXAA_MOUSE NM_008264 HXAD_MOUSE # The following 2 lines demonstrate multiple data values for 1 key: AK083575 NM_010449 AK083575 Q8BNI8 M95599 NM_010451 M28021 HXA5_MOUSE NM_010454 HXA6_MOUSE 
/goldenPath/help/covidBrowserIntro.html:COVID_Browser_Intro	Introduction to the SARS-CoV-2 Genome Browser The UCSC Genome Browser is an open-source, interactive sequence visualization tool that has been a cornerstone of genomics since we released the first human genome assembly 20 years ago. Cited in more than 37,000 scientific articles and used by thousands of researchers each day; it allows for cross-referencing of research, clinical, and epidemiology data against reference genomes, including SARS-CoV-2. This data is continuously updated and added to as new datasets become available. For a more thorough description, please reference our SARS-CoV-2 Genome Browser Nature Genetics paper. We also post updates and COVID Browser resources to out COVID-19 Browser home page. This guide will go through some of the most important use cases of the SARS-CoV-2 Genome Browser. These topics include: - Orientation and Navigation - Gene Data and Sequence Alignments - Variation and Immunology data - Phylogenetic Contact Tracing using UShER - Other tools and data downloads - Support and Collaboration For those who prefer a video explanation, we also have the following tutorial: Genome Browser Orientation and Navigation The standardized reference genome displayed on the COVID Genome Browser is from one of the first isolated cases, known as NC_045512v2 or wuhCor1. With more than 80 track datasets across the SARS-CoV-2 reference genome's nearly 30,000 RNA bases, navigation is essential to finding the information you want to see. Below is an example view of the SARS-CoV-2 Genome Browser with labeled sections highlighting the navigation, reference sequence, annotations, and other available track datasets. Navigation controls at the top allow users to move left and right and to zoom. The search box allows users to search for particular features or to move to exact genomic coordinates. The RNA sequence is shown at the top only when the view is sufficiently zoomed in. Annotations are shown for data tracks that have been set to visible in the available tracks section at the bottom. Tracks can be configured with a right-click or by clicking on their name near the bottom of the page. [Labeled orientation to the Genome Browser] This is a view of the SARS-CoV-2 Genome Browser (COVID Browser) with labeled elements to help with orientation. Interact with this session by clicking on the picture. To read the full caption, please go to our Nature Genetics paper. Genes and Sequence Alignments Gene and protein annotations are organized by the contributor, most notably NCBI and UniProt. Having multiple information sources allows a consensus to be formed among datasets. Like many viral genomes, molecular complexity arises from polyproteins rearranging, generating ~29 protein products. Most notable among these is the S (spike) protein which defines coronaviruses and allows entry into cell membrane. Additional tracks contain information such as interactions between viral proteins and human proteins (protein interact), PDB structures, and RNA structure annotations (Rangan RNA), and more. Sequence alignments and conservation data are also available across the SARS-CoV-2 genome, from large-scale views to individual bases and amino acids. Four conservation tracks compare sequences with 44 bat coronaviruses, 119 vertebrate coronaviruses, 7 human coronaviruses, and PhyloCSF computed conservation scores. The tracks display differently depending on visibility mode and the number of bases on the screen. Datasets can be turned on by setting the dropdown next to the data track name from "hide" to dense, squish, pack, or full. Then click the refresh button to see these changes in effect. Clicking on a data track name will take you to a description with more information on the dataset, display conventions, methods, and references. Clicking on a particular item will take you to a page with complete information about that item and dataset. [Some of the gene and conservation data on the Genome Browser] This Genome Browser display shows some of the gene and conservation tracks available on the SARS-CoV-2 genome. You should be able to see UniProt protein products, regions of interest, and domains all mapped against the SARS-CoV-2 genome. Below those tracks are two different conservation alignments in "squish" and "pack" formats, comparing bat-host and human-host coronavirus sequences with the reference SARS-CoV-2 genome. Interact with this session by clicking on the picture. Exploring Variation and Immunology Data The SARS-CoV-2 Genome Browser displays data on variation within SARS-CoV-2 from UniProt, GenBank, GISAID, Nexstrain, and other providers. These datasets cover global trends in SARS-CoV-2 variation among all available public sequences, with regional descriptions available through clicking into a particular entry. A few of the most notable tracks under the "Variation and Repeats" section are the Phylogeny: Public track, which shows a continuously updating phylogenetic tree that clusters similar sequences, with the frequency of each mutation shown by the height of the bar at that particular base. Tools are provided to filter these data to show only well-supported mutation calls, set thresholds for minor-allele frequency, and display data for specific clades. Another track is the spike protein mutations from community annotations, highlighted as amino acid changes with red indicating strong antibody escape in receptor-binding domain (RBD) mutation screens. The Genome Browser has also has the Variants of Concern track, which pinpoints each accumulated mutation that defines 4 strains of SARS-CoV-2 of particular concern, labeled based on lay terms (such as 'California variant') as well as the using the lineage defined by the Pangolin software (such as 'B.1.1.7'). The Genome Browser also provides 12 immunology datasets that can inform potential therapeutic targets or public health risks. Protein epitopes are highlighted in the genome by multiple tracks, including those from the Immune Epitope Database (IEDB) and from a study of COVID+ patients. Of particular interest are the datasets describing surveys of antibody response across a variety of SARS-CoV-2 variants in the receptor-binding domain (Antibody Escape Mutations). [Some of the variation and immunology data on the Genome Browser] This image shows some of the variation data tracks that can be displayed on the SARS-CoV-2 genome, specifically zoomed into the receptor-binding domain of the Spike protein. Validated epitopes are displayed in black that may be a target for therapeutic antibodies. In red and black, antibody escape scores are are shown for each genome position. Smaller tick marks show amino acid or nucleotide changes from different sources, with more information available by clicking into the item. Genetic Contact Tracing with UShER The UCSC Genome Browser has developed a tool that allows placement of SARS-CoV-2 sequences onto existing phylogenetic trees far faster than previous methods, allowing instantaneous tracing of strains and transmission events. This tool is called Ultrafast Sample placement on Existing tRees (UShER) and exists as an interactive web-tool to compare sequences and link to existing public phylogenetic trees. [Example of the UShER phylogeny placement tool] After uploading a Fasta file, the tool returns a page with quality metrics such as: number of bases aligned, number of Ns, and number of maximally parsimonious placements along with the lineage and clade of the nearest neighbor. Colored boxes highlight possible quality issues, green meaning this was a high confidence placement. SARS-CoV-2/ COVID Phylogenic Trees You can view your aligned SARS-CoV-2 sequence genotypes along with their closest known relatives among the 150,000+ public sequences. You can compare among your uploaded samples or trace possible transmission vectors using mutational signatures. [Example of the UShER phylogeny placement tool tree features] The uploaded sequences are highlighted in blue alongside their most closely aligned public sequences. You can investigate genotypes and relationships between samples. Other tools, downloads, and features Custom Tracks, BLAT, Track Hubs Along with a suite of data tracks, filters, and visualization options for the SARS-CoV-2 genome, the UCSC Genome Browser offers many additional ways to interface with our data. You can upload your data on the reference genome in nearly any format with our Custom Track tool. If you have unaligned sequence, you can use our BLAT sequence alignment tool to get coordinates and base-by-base comparison with any reference genome. We also display formatted data as Track Hubs and curate a list of user-submitted Public Track Hubs. Downloads, Table Browser, JSON API, SQL As part of our open-source, open-access philosophy, we try to make it as easy as possible for researchers to download entire datasets or filtered subsets. Each track description page has a Data Access section which points users to our main options for data download. For downloading complete datasets, our SARS-CoV-2 download directory provides access to all our source files for transparency and reproducibility. Our Table Browser tool lets users interact with our data using a variety of filters based on score, identifiers, or any other field. Table Browser also allows users to convert data into multiple different formats (e.g. BED, GTF) and to access different formatted sequence outputs (in FASTA format). We have a JSON API which can be programmatically called and return any dataset in its entirety or as a filtered subset based on documented input parameter. We also offer a Public SQL server for similar flexible, automatic way to access genomic data and annotations. Along with this particular virus genome browser, we have thousands of genomes available for visualization and analysis from our genome assemblies gateway page. Support and Collaboration The Genome Browser offers rapid email support for anything related to our tools. If your question is general or may have been asked before, please review our Browser documentation and our archive of previously answered questions. If you would still like help, please go to our Contact Us page to see access our email support. When contacting us, please include a session link, images, and example data if applicable. We are active on social media, you can follow us on Twitter or Facebook. We are always looking to collaborate with researchers and add new datasets to our site. We also seek to continuously improve our tools to meet the needs of the scientific community. If you have any collaboration ideas, contributions, or feature requests, please reach out through our suggestion page. 
/goldenPath/help/maf.html:Genome_Browser_maf_Format	Redirect 
/goldenPath/help/cram.html:Genome_Browser_CRAM_Format	CRAM Track Format The UCSC Genome Browser is capable of displaying both the BAM and CRAM file formats. While BAM files contain all sequence data within a file, CRAM files are smaller by taking advantage of an additional external "reference sequence" file. This file is needed to both compress and decompress the read information. Since CRAM files are more dense than BAM files, many groups are switching to the CRAM format to save disk space. For CRAM tracks to load there is an expectation that the checksum of the reference sequence used to create the CRAM will be in the CRAM header. A file with a matching checksum is also expected to be accessible from the EBI RefGet CRAM reference registry (see References for CRAM resources). Otherwise, users must specify a refUrl setting that will point to a server that is offering up the reference sequences (see Example Four). Since the loading of CRAM data requires the specific reference sequence used to create the CRAM file, it is very important that the exact same reference sequence is used for compression and decompression. When a CRAM file is first loaded on a given chromosome, a check for the preexistence in a special browser "cramCache" directory of the specified reference checksum will take place. If the reference sequence information specific for that CRAM for the currently viewed chromosome region does not exist, a message will display about the file not being found along with a note about downloading the reference from the EBI CRAM reference registry if it is available or from another Refget server using the refUrl setting. A refresh of the page once the download is complete will display the CRAM data as if it were a BAM file. The track lines to describe CRAM tracks are identical to track lines for BAM tracks. This includes the type parameter, which is still bam even for CRAM tracks. The only difference is that instead of providing the URL to a BAM file, the URL instead points to a CRAM file. Please also note that just as a BAM file requires an associated BAM.bai index file, a CRAM file will require an associated CRAM.crai index file in the same location to load. Example #1 Here is an example CRAM track that displays around the gene SOD1 on hg19 that can be cut and pasted as text into the Custom Tracks page: track type=bam db=hg19 name=exampleCRAM bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/cramExample.cram Please note at the above URL location there is also a http://genome.ucsc.edu/goldenPath/help/examples/cramExample.cram.crai file. If this .crai file is at a different URL, the bigDataIndex=<URL> option must be added. Clicking this following link will also load the above track. The information following hgct_customText is equivalent to pasting the text in to the Custom Tracks page: http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg19&position=chr21%3A33031597-33041570&hgct_customText=track%20type=bam%20name=exampleCRAM%20bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/cramExample.cram Example #2 If the URL to a CRAM file ends with .cram, you can paste the URL directly into the custom track management page, click submit and view it in the Browser. The track name will then be the name of the file. If you want to configure the track name and descriptions, you will need to create a track line, as shown in the above example. Learn more about track line options and configuring custom tracks here. Here is an example URL to a CRAM file from the 1000 Genomes Project that can be pasted directly into the Custom Tracks page: ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/phase3/data/HG00096/exome_alignment/HG00096.mapped.ILLUMINA.bwa.GBR.exome.20120522.bam.cram You can see by adding the above link the Browser automatically assigns the type=bam and the track name=HG00096.mapped.ILLUMINA.bwa.GBR.exome.20120522.bam to the created track to browse. Clicking the following image will load a CRAM file from the 1000 Genomes Project. [] This CRAM display takes advantage of using the new "density graph" feature where the bam.cram reads are displayed as a bar graph by checking the box next to "Display data as a density graph" on the Custom Track Settings page. Example #3 The CRAM format is also supported in track hubs. Below is an example trackDb.txt stanza that would display a CRAM files from the 1000 Genomes Project. To learn more about using Track Hubs see the User Guide and associated Quick Start Guides to building hubs. Note that type bam is used to display CRAM files in hubs, just as type bam is used in custom CRAM tracks. track cram61 type bam shortLabel HG00361 longLabel This CRAM file is from the 1000 Genomes Project HG00361 visibility pack bigDataUrl ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/phase3/data/HG00361/exome_alignment/HG00361.mapped.ILLUMINA.bwa.FIN.exome.20120522.bam.cram Example #4 For genomes that are not registered in the EBI CRAM Reference Registry, the refUrl setting is used to point the browser to the appropriate place to find the reference sequence. The refUrl setting is used with the URL of the reference server, such as refUrl http://university.edu/URL/cramRef/%s where the %s gets replaced by the RefGet MD5 checksum which identifies the reference sequence. The example below shows a hub track stanza using the refUrl setting: track cramExample type bam visibility full shortLabel cramExRefUrl longLabel This CRAM file points to a reference sequence specified by refUrl refUrl http://university.edu/URL/cramRef/%s bigDataUrl ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/phase3/data/HG00096/alignment/HG00096.mapped.ILLUMINA.bwa.GBR.low_coverage.20120522.bam.cram The use of refUrl can also be employed on a custom track line: track type=bam db=hg19 name=cramExRefUrl refUrl=http://university.edu/URL/cramRef/%s bigDataUrl=ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/phase3/data/HG00096/alignment/HG00096.mapped.ILLUMINA.bwa.GBR.low_coverage.20120522.bam.cram References Below is a collection of helpful CRAM resources: - CRAM toolkit - CRAM reference registry - Refget: standardized access to reference sequences - CRAM format specification (version 3.0) - Using CRAM within Samtools - International Genome Sample Resource (IGSR) CRAM Tutorial - Dave Tang blog post about "BAM to CRAM" Sharing your data with others If you would like to share your CRAM data track with a colleague, learn how to create a URL by looking at Example 11 on this page. Activating CRAM support for the Genome Browser To find documentation on how to set up CRAM support on a mirror of the UCSC Genome Browser please see this following README.cram file. 
/goldenPath/help/image.html:Genome_Browser_Tracks_Image	Help with the browser tracks image If you are having difficulties viewing the Genome Browser, it may be a result of a recent update to our software. To clear up the tracks image, you can force a refresh/reload of the page in your Internet browser. Depending on which Internet browser you are using, you may need to hold down the Shift key while simultaneously clicking the "reload" button (or, in some browsers, Ctrl+R works). Once you have refreshed the page, the tracks image should display correctly. Sorry for any inconvenience. We hope that you understand that the website is continually evolving so that we can provide you with more features and data. 
/goldenPath/help/trix.html:Genome_Browser_Trix_Indices	Trix Indices A Trix index consists of a pair of files that allow for fast look-up of free text associated with a list of identifiers. The index is created from a single line-oriented text file using the program ixIxx. Each line in the text file starts with an identifier, followed by free text associated with the ID. The search is not case sensitive, so any case-combination of the free text entered will be matched. For a more complete description of how to make a searchable track hub (or custom track), please visit the Quick Start Guide to Searchable track hubs. To complete the steps below you must first download the ixIxx utility. For more information on downloading our command line utilities, see these instructions. Example 1 To create a Trix index, follow these steps: 1. Prepare a text file that associates your IDs with free text: id1 this is text for id1 id2 this is text for id2 id3 this is text for id3 2. Run the ixIxx program on your text file. ixIxx input.txt myTrix.ix myTrix.ixx Example 2 1. If you have a bigBed track with unique gene names such as SIRT1, BRCA1, TP53 in the fourth name column, and you built the bigBed using the option of -extraIndex=name to index the name field you can create an input.txt such as the following that associates the bigBed name with other identifiers people might search for or with shorter spellings of the names: SIRT1 sirt1 sir sirt Sirtuin SIR2-Like ENSG00000096717 NM_012238 BRCA1 brca1 brca brc breast cancer 1 ENSG00000012048 NM_007300 TP53 tp53 tp5 Tumor Protein P53 ENSG00000141510 NM_001126112 2. You would then run the ixIxx program on your text file, taking into account the length of your longest word. ixIxx input.txt myTrix.ix myTrix.ixx The ixIxx utility has a default of 31 characters for words. If you build an input.txt with longer words, such as very long accessions, you neeed to add the option -maxWordLength=N to override the default and expand it to the size you are using for the longest words in your index. Resources and examples If you want to use your Trix index in a track hub, see the searchTrix setting in the Track Database Definition Document. Review our Quick Start Guide to Searchable track hubs for illustrated steps building a track hub. There are also tools available for taking genePred format to trix format, such as this gpToIx.pl perl script. 
/goldenPath/help/bigMaf.html:Genome_Browser_bigMaf_Alignment_Format	bigMaf Track Format The bigMaf format stores multiple alignments in a format compatible with MAF files, which is then compressed and indexed as a bigBed. The bigMaf files are created using the program bedToBigBed, run with the -as option to pull in a special autoSql (.as) file that defines the fields of the bigMaf. The bigMaf files are in an indexed binary format. The main advantage of this format is that only those portions of the file needed to display a particular region are transferred to the Genome Browser server. Because of this, bigMaf files have considerably faster display performance than regular MAF files when working with large data sets. The bigMaf file remains on your local web-accessible server (http, https or ftp), not on the UCSC server, and only the portion needed for the currently displayed chromosomal position is locally cached as a "sparse file". If you do not have access to a web-accessible server and need hosting space for your bigMaf files, please see the Hosting section of the Track Hub Help documentation. bigMaf file definition The following autoSql definition is used to specify bigMaf multiple alignment files. This definition, contained in the file bigMaf.as, is pulled in when the bedToBigBed utility is run with the -as=bigMaf.as option. bigMaf.as table bedMaf "Bed3 with MAF block" ( string chrom; "Reference sequence chromosome or scaffold" uint chromStart; "Start position in chromosome" uint chromEnd; "End position in chromosome" lstring mafBlock; "MAF block" ) An example: bedToBigBed -type=bed3+1 -as=bigMaf.as -tab bigMaf.txt hg38.chrom.sizes bigMaf.bb Supporting frame and summary definitions Alongside the bigMaf file, two other summary and frame bigBeds are created. The following autoSql definition is used to create the first file, pointed to online with summary <url>, rather than the standard bigDataUrl <url> used with bigMaf. The file mafSummary.as, is pulled in when the bedToBigBed utility is run with the -as=mafSummary.as option. mafSummary.as table mafSummary "Positions and scores for alignment blocks" ( string chrom; "Reference sequence chromosome or scaffold" uint chromStart; "Start position in chromosome" uint chromEnd; "End position in chromosome" string src; "Sequence name or database of alignment" float score; "Floating point score." char[1] leftStatus; "Gap/break annotation for preceding block" char[1] rightStatus; "Gap/break annotation for following block" ) An example, bedToBigBed -type=bed3+4 -as=mafSummary.as -tab bigMafSummary.bed hg38.chrom.sizes bigMafSummary.bb. Another tool, hgLoadMafSummary generates the input bigMafSummary.bed file. The following autoSql definition is used to create the second file, pointed to online with frames <url>. The file mafFrames.as, is pulled in when the bedToBigBed utility is run with the -as=mafFrames.as option. mafFrames.as table mafFrames "codon frame assignment for MAF components" ( string chrom; "Reference sequence chromosome or scaffold" uint chromStart; "Start range in chromosome" uint chromEnd; "End range in chromosome" string src; "Name of sequence source in MAF" ubyte frame; "frame (0,1,2) for first base(+) or last bast(-)" char[1] strand; "+ or -" string name; "Name of gene used to define frame" int prevFramePos; "target position of the previous base (in transcription direction) that continues this frame, or -1 if none, or frame not contiguous" int nextFramePos; "target position of the next base (in transcription direction) that continues this frame, or -1 if none, or frame not contiguous" ubyte isExonStart; "does this start the CDS portion of an exon?" ubyte isExonEnd; "does this end the CDS portion of an exon?" ) An example, bedToBigBed -type=bed3+8 -as=mafFrames.as -tab bigMafFrames.txt hg38.chrom.sizes bigMafFrames.bb. Another tool, genePredToMafFrames generates the input bigMafFrames.txt file. Note that the bedToBigBed utility uses a substantial amount of memory: approximately 25% more RAM than the uncompressed BED input file. Creating a bigMaf track To create a bigMaf track, follow these steps: Step 1. If you already have a MAF file you would like to convert to a bigMaf, skip to Step 3. Otherwise, download this example MAF file for the human GRCh38 (hg38) assembly. Step 2. If you would like to include optional reading frame and block summary information, download the chr22_KI270731v1_random.gp genePred file. Step 3. Download the autoSql file bigMaf.as needed by bedToBigBed. If you have opted to include the optional frame summary and information with your bigMaf file, you must also download the autoSql files mafSummary.as and mafFrames.as files. Here are wget commands to obtain the above files and the hg38.chrom.sizes file mentioned below: wget https://genome.ucsc.edu/goldenPath/help/examples/chr22_KI270731v1_random.maf wget https://genome.ucsc.edu/goldenPath/help/examples/chr22_KI270731v1_random.gp wget https://genome.ucsc.edu/goldenPath/help/examples/bigMaf.as wget https://genome.ucsc.edu/goldenPath/help/examples/mafSummary.as wget https://genome.ucsc.edu/goldenPath/help/examples/mafFrames.as wget http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.chrom.sizes Step 4. Download the bedToBigBed and mafToBigMaf programs from the UCSC binary utilities directory. If you have opted to generate the optional frame and summary files for your multiple alignment, you must also download the hgLoadMafSummary, genePredSingleCover, and genePredToMafFrames programs from the same directory. Step 5. Use the fetchChromSizes script from the same directory to create a chrom.sizes file for the UCSC database with which you are working (e.g., hg38). Alternatively, you can download the chrom.sizes file for any assembly hosted at UCSC from our downloads page (click on "Full data set" for any assembly). For example, the hg38.chrom.sizes file for the hg38 database is located at http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.chrom.sizes. mafToBigMaf hg38 chr22_KI270731v1_random.maf stdout | sort -k1,1 -k2,2n > bigMaf.txt bedToBigBed -type=bed3+1 -as=bigMaf.as -tab bigMaf.txt hg38.chrom.sizes bigMaf.bb Note that the hg38 in the mafToBigMaf hg38 command indicates the referenceDb and matches the expected prefix of the primary species' sequence name, for instance hg38 for the hg38.chr22_KI270731v1_random found in the input example chr22_KI270731v1_random.maf file. Step 6. Follow the below steps to create the binary indexed mafFrames and mafSummary files to accompany your bigMaf file: genePredSingleCover chr22_KI270731v1_random.gp single.gp genePredToMafFrames hg38 chr22_KI270731v1_random.maf bigMafFrames.txt hg38 single.gp bedToBigBed -type=bed3+8 -as=mafFrames.as -tab bigMafFrames.txt hg38.chrom.sizes bigMafFrames.bb hgLoadMafSummary -minSeqSize=1 -test hg38 bigMafSummary chr22_KI270731v1_random.maf cut -f2- bigMafSummary.tab | sort -k1,1 -k2,2n > bigMafSummary.bed bedToBigBed -type=bed3+4 -as=mafSummary.as -tab bigMafSummary.bed hg38.chrom.sizes bigMafSummary.bb Step 7. Move the newly created bigMaf file (bigMaf.bb) to a web-accessible http, https or ftp location. If you generated the bigMafSummary.bb and/or bigMafFrames.bb files, move those to a web accessible location, likely same location as the bigMaf.bb file. Step 8. Construct a custom track using a single track line. Note that any of the track attributes listed here are applicable to tracks of type bigBed. The most basic version of the track line will look something like this: track type=bigMaf name="My Big MAF" description="A Multiple Alignment" bigDataUrl=http://myorg.edu/mylab/bigMaf.bb summary=http://myorg.edu/mylab/bigMafSummary.bb frames=http://myorg.edu/mylab/bigMafFrames.bb Step 9. Paste the custom track line into the text box on the custom track management page. Navigate to chr22_KI270731v1_random to see the example data for this track. The bedToBigBed program can be run with several additional options. For a full list of the available options, type bedToBigBed (with no arguments) on the command line to display the usage message. Examples Example #1 In this example, you will create a bigMaf custom track using an existing bigMaf file, bigMaf.bb, located on the UCSC Genome Browser http server. This file contains data for the hg38 assembly. To create a custom track using this bigMaf file: 1. Construct a track line that references the file: track type=bigMaf name="bigMaf Example One" description="A bigMaf file" bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigMaf.bb frames=http://genome.ucsc.edu/goldenPath/help/examples/bigMafFrames.bb summary=http://genome.ucsc.edu/goldenPath/help/examples/bigMafSummary.bb 2. Paste the track line into the custom track management page for the human assembly hg38 (Dec. 2013). 3. Click the "submit" button. Note that additional track line options exist that are specific to the MAF format. For instance, adding the parameter setting speciesOrder="panTro4 rheMac3 mm10 rn5 canFam3 monDom5" to the above example will specify the order of sequences by species. Custom tracks can also be loaded via one URL line. This link loads the same bigMaf.bb track and sets additional display parameters in the URL: http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&position=chr22_KI270731v1_random&hgct_customText=track%20type=bigMaf%20name=Example%20bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigMaf.bb%20visibility=pack After this example bigMaf is loaded in the Genome Browser, click into an alignment on the browser's track display. Note that the details page displays information about the individual alignments, similar to that which is available for a standard MAF track. Example #2 In this example, you will create a bigMaf file from an existing bigMaf input file, bigMaf.txt, located on the UCSC Genome Browser http server. 1. Save the bed3+1 example file, bigMaf.txt, to your computer (Step 6, above). 2. Save the autoSql file bigMaf.as to your computer (Step 3, above). 3. Download the bedToBigBed utility (Step 4, above). 4. Save the hg38.chrom.sizes text file to your computer. This file contains the chrom.sizes for the human (hg38) assembly (Step 5, above). 5. Run the bedToBigBed utility to create a binary indexed MAF file (Step 6, above): bedToBigBed -type=bed3+1 -tab -as=bigMaf.as bigMaf.txt hg38.chrom.sizes bigMaf.bb 6. Move the newly created bigMaf file (bigMaf.bb) to a web-accessible location (Step 7, above). 7. Construct a track line that points to the bigMaf file (Step 8, above). 8. Create the custom track on the human assembly hg38 (Dec. 2013), and view it in the Genome Browser (step 9, above). Sharing your data with others If you would like to share your bigMaf data track with a colleague, learn how to create a URL by looking at Example 6 on this page. Extracting data from the bigMaf format Because bigMaf files are an extension of bigBed files, which are indexed binary files, it can be difficult to extract data from them. UCSC has developed the following programs to assist in working with bigBed formats, available from the binary utilities directory. - bigBedToBed — converts a bigBed file to ASCII BED format. - bigBedSummary — extracts summary information from a bigBed file. - bigBedInfo — prints out information about a bigBed file. As with all UCSC Genome Browser programs, simply type the program name (with no parameters) at the command line to view the usage statement. Troubleshooting If you encounter an error when you run the bedToBigBed program, check your input file for data coordinates that extend past the the end of the chromosome. If these are present, run the bedClip program (available here) to remove the problematic row(s) in your input file before running the bedToBigBed program. 
/goldenPath/help/hgBamTrackHelp.html:Genome_Browser_BAM_Configuration	Configuring BAM tracks Genome Browser BAM tracks may be configured in a variety of ways to highlight different aspects of the displayed information. The configuration options are described here and related to custom track settings that can alter the default appearance of the custom track. Click here for more information on BAM custom track creation. - Attempt to join paired end reads by name: This checkbox appears only if pairEndsByName is included in the track settings. When checked, SAM/BAM records with the same name will be joined into pairs for display, with a line drawn between them. - Minimum alignment quality: Exclude alignments with quality less than the given number. The default is 0, unless changed by the track setting minAliQual. - Color track by bases: By default, mismatching bases are highlighted in the display. Change the selection to "item bases" to see all base values from the query sequence, or "OFF" to ignore query sequence. - Additional coloring modes: Other aspects of the alignments can be displayed in color or grayscale. The default mode is "Color by strand" (bamColorMode=strand), unless the bamColorMode track setting specifies gray, tag or off. - Color by strand: alignments on the reverse strand are colored dark red, alignments on the forward strand are colored dark blue. - Grayscale: items are shaded according to the chosen method: alignment quality, base qualities, or unpaired ends. The alignment qualities of items are shaded on a scale of 0 (lightest) to 99 (darkest). Base qualities are shaded on a scale of 0 (lightest) to 40 (darkest). When "unpaired ends" is selected, items that were paired in sequencing but whose mate was not mapped are colored gray, while singletons and properly paired items are black. Alignment quality is the default (bamGrayMode=aliQual) unless bamGrayMode track setting is baseQual or unpaired. - Use R,G,B colors specified in user-defined tag: SAM/BAM may include user-defined tags, whose names begin with X, Y or Z and include one other letter or number. The user-defined tag named here specifies red, green and blue (RGB) intensities as a zero-terminated string (tag type Z) containing comma-separated triples of numbers from 0-255. For example, if a SAM/BAM record includes the tag YC:Z:255,0,0, then the item is colored red; YC:Z:0,0,255 makes the item blue. By default, the tag is "YC" unless changed using the track setting bamColorTag. - No additional coloring - Display data as a density graph: This feature enables the BAM data to be displayed as a bar graph where the height is proportional to the number of reads mapped to each genomic position. Through dynamic calculation of items in the current window, this feature plots a line similar to a wiggle graph that can be customized with a number of graph-based configuration options such as drawing indicator lines, smoothing plots, adjusting graph height and vertical range, and switching from bars to points. Please note that the feature is best displayed with Display mode set to full and that the default Data view scaling is auto-Scale to data view. Also, please note that when set to display as a density graph, other BAM display options described on this page, such as coloring for strand, alignment quality and base qualities, are not applied. When you have finished making your configuration changes, click the Submit button to return to the annotation track display page. 
/goldenPath/help/barChart.html:barChart_and_bigBarChart_Track_Format	barChart and bigBarChart Track Format The barChart (and bigBarChart) track format displays a graph of category-specific values over genomic regions, similar to the GTEx Gene track. This format is useful for displaying gene expression and other datasets where it is desirable to compare a set of variables over genomic regions. While a barChart track can effectively show datasets with single values for each variable (e.g. comparing individual samples), the format provides specific features to display studies comprised of a large set of samples for each variable (e.g. comparing tissues with multiple samples for each tissue). In this usage, the main genome browser display presents a graph of summary values (e.g. medians) for each variable, and the distribution of sample values across variables is shown via a boxplot graph shown on the details page for each region. The barChart format is available as a standalone plain text bed6+ format for use with smaller datasets as a custom track, and as a binary indexed format (bigBarChart) suitable for track hubs and custom tracks. The bigBarChart format provides more track customization features (i.e. schema customization, and label configuration support), and is recommended for users who can use command-line tools and have web-accessible data storage. If you do not have web-accessible data storage, please see the Hosting section of the Track Hub Help documentation. barChart format files are converted to bigBarChart files using the program bedToBigBed, run with the -as option to pull in a special autoSql (.as) schema file defining the fields of the bigBarChart. Below is an example of the barChart format in 'full' visibility mode [BarChart example in full mode] The 'squish' display mode draws one colored rectangle indicating the category (e.g. tissue) with highest value of the measured metric (e.g. gene expression) if it contributes more than 10% to the total expression, otherwise the chart is colored black. The following image shows the GTEx Genes track in 'squish' mode; the beige colored item (tissue) has the highest expression in the ACE2 gene and represents more than 10% of total expression. Click into the colored rectange for more information. [BarChart example in squish mode] barChart format definition The following autoSql definition illustrates the basic schema supporting barChart (and bigBarChart) tracks. table bigBarChart "bigBarChart bar graph display" ( string chrom; "Reference sequence chromosome or scaffold" uint chromStart; "Start position in chromosome" uint chromEnd; "End position in chromosome" string name; "Name or ID of item" uint score; "Score (0-1000)" char[1] strand; "'+','-' or '.'. Indicates whether the query aligns to the + or - strand on the reference" string name2; "Alternate name of item" uint expCount; "Number of bar graphs in display, must be <= 100" float[expCount] expScores; "Comma separated list of category values." bigint _dataOffset; "Offset of sample data in data matrix file, for boxplot on details page, optional only for barChart format" int _dataLen; "Length of sample data row in data matrix file, optional only for barChart format" ) Column Explanations The first 6 fields of the barChart format are the same as the first 6 fields of the standard BED format. The name2 field provides an alternate item name, useful if you would like to associate multiple transcripts to a single gene locus, different variables to the same experiment type, etc. The expCount and expScores fields are used as in the Microarray format; they define the number of categories and a value for each category (see example #1 below). The _dataOffset and _dataLen fields are used internally by the track to locate sample values for a region in an optional matrix file containing all sample values. These values are used to draw a boxplot of all sample data on the details page for the bar chart. When a matrix file is not supplied, these fields should be set to 0. (As a convenience, these fields are optional for barChart custom tracks). When creating bigBarChart files, we encourage you to customize the title and field descriptions of the prototype autoSql schema to better describe your data. In the example below, the name field of the track refers to a transcript, while the name2 field represents a gene: table xyzGeneExpression "XYZ gene expression barChart" ( string chrom; "Reference sequence chromosome or scaffold" uint chromStart; "Start position in chromosome" uint chromEnd; "End position in chromosome" string name; "Transcript name" uint score; "Score (0-1000), derived from total expScores (below)" char[1] strand; "+, -, or ., indicating orientation of the item" string name2; "Gene name" uint expCount; "Number of tissues" float[expCount] expScores; "Comma separated list of median expression in RPKM for each tissue." bigint _dataOffset; "Offset of sample data in data matrix file" int _dataLen; "Length of sample data row in data matrix file" ) Customing this file will make your data more easily interpreted by users, who will see the field descriptions when accessing the track data from the Table Browser, when viewing items on the Genome Browser details pages (via the "view table schema" link), and (for users who download files), from the -as option of the bigBedInfo tool. Creating barChart and bigBarChart custom tracks The steps for creating barChart tracks differ from the process for creating bigBarChart tracks. The steps also differ based on whether you have an input matrix file (generated perhaps from an RNA-Seq differential expression analysis pipeline) or not. If you have an expression matrix-like file, skip to Example #3, otherwise follow example 1 below. Example #1 In this example, you will create a barChart custom track using example bed6+3 data. 1. Paste the following track line into the custom track management page for the human assembly hg38. track type=barChart name="barChart Example One" description="A barChart file" barChartBars="adiposeSubcut breastMamTissue colonTransverse muscleSkeletal wholeBlood" visibility=pack browser position chr14:95,081,796-95,436,280 # chrom chromStart chromEnd string name score strand name2 expCount expScores _dataOffset _dataLen chr14 95086227 95158010 DICER1 999 - ENSG00000100697.10 5 2.94,11.60,38.00,6.69,4.89 chr14 95181939 95319906 CLMN 999 - ENSG00000165959.7 5 7.08,69.53,9.32,1.38,1.68 chr14 95417493 95475836 SYNE3 999 - ENSG00000176438.8 5 7.29,3.73,0.74,20.35,1.39 2. Click the "submit" button. After the file loads in the Genome Browser, you should see an automatically colored bar graph with 5 bars. Hovering the mouse over any of the individual bars will display the name of the particular bar ("wholeBlood", "adiposeSubcut", ...) as well as the value associated with that bar (10.94, 0.74, ...). The order of bar names in the barChartBars field of track line should exactly match the order of the values in the expScores field. Example #2 In this example, you will create a bigBarChart track out of an existing bigBarChart format file, located on the UCSC Genome Browser http server. This file contains data for the hg38 assembly. To create a custom track using this file: 1. Construct a track line referencing the file: track type=bigBarChart name="bigBarChart Example One" description="A bigBarChart file" barChartBars="adiposeSubcut breastMamTissue colonTransverse muscleSkeletal wholeBlood" visibility=pack bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/barChart/hg38.gtexTranscripts.bb browser position chr14:95,081,796-95,436,280 2. Paste the track line into the custom track management page for the human assembly hg38. 3. Click the "submit" button. After the file loads in the Genome Browser, you should see an automatically colored bar graph with 5 bars. The same rules apply to bigBarChart custom tracks as barChart custom tracks in that the order of names in the barChartBars field should exactly match the order of values from the expScores field in the bigBarChart file. Example #3 In this example, you will use the helper scripts expMatrixToBarchartBed and bedJoinTabOffset on example matrix and category files in order to generate a bed6+5 barChart format file, which can be loaded as a custom track into the Genome Browser. The matrix file is a tab-separated (must be tabs, not spaces) file of the following form, perhaps resulting from an RNA-Seq analysis pipeline. Please note that the first line must describe each column as in the example snippet below. transcript sample1 sample2 sample3 sample4 sample5 ... transcriptName value1 value2 value3 value4 value5 ... The categories file then provides more meta information about this matrix file. It is a two column, tab-separated file that maps the samples in the matrix file to a specific category: sample1 category1 sample2 category1 ... ... sampleA category2 sampleB category2 ... ... sampleX category3 sampleY category3 ... ... Each column in the first line of the matrix file must be found in the categories file. We have provided an example category file and matrix file to follow along with the rest of this example. To create a custom track in this form, follow the below steps: 1. Create a bed6+1 file to use as a map for the items in your matrix (does not need to be all-inclusive). This file, with example lines such as: chr14 95086227 95158010 ENSG00000100697.10 999 - DICER1 will be one of three input files to the helper script expMatrixToBarchartBed. For an example file to follow along with the rest of this example, you can download an example bed6+1 file here. 2. Download the helper programs expMatrixToBarchartBed and bedJoinTabOffset from the utilities directory appropriate for your operating system. 3. Make sure the programs downloaded above are in your system's PATH variable. For example, if you downloaded the programs to your $HOME/Downloads directory, set your PATH variable accordingly: export PATH=$PATH:$HOME/Downloads 4. Now run expMatrixToBarchartBed (which in turn runs bedJoinTabOffset) like so: expMatrixToBarchartBed categoriesFile matrixFile bedInputFile outputBed The argument outputBed will be a bed6+5 file, with the expCount, expScores, _dataOffset, and _dataLen fields computed for you, for example: chr14 95086227 95158010 ENSG00000100697.10 999 - DICER1 5 10.94,11.60,8.00,6.69,4.89 93153 26789 expMatrixToBarchartBed will also output the order of the scores in the expScores field, which you can then copy and paste into the barChartBars field of the custom track line so the bars displayed in the browser match the right values: The columns and order of the groups are: #chr start end name score strand name2 expCount expScores;adiposeSubcut breastMamTissue colonTransverse muscleSkeletal wholeBlood _offset _lineLength If you have already pre-computed expCount and expScores, and just need offsets into your matrix file for a more descriptive details page, run only bedJoinTabOffset like so: bedJoinTabOffset matrixFile exampleBed6+3 outBed 5. Now that we have a bed6+5 format file that corresponds to the barChart format, we can construct a track line and prepend it to our bed6+5 file so the Genome Browser will recognize it: track type=barChart name="barChart Example" description="A barChart file" barChartBars="adiposeSubcut breastMamTissue colonTransverse muscleSkeletal wholeBlood" barChartMetric=median" visibility=pack 6. Use the upload button on the custom track management page for the human assembly hg38 to upload the file just created. 7. Click the "submit" button. expMatrixToBarchartBed automatically computes the median values for all the samples in the matrix file, which is useful when your experiment contains data from 8000 samples (such as the GTEx data). Furthermore, expMatrixToBarchartBed can compute the mean value of all samples in a category of the matrix file (instead of the default median), in addition to allowing for a specific ordering of the expScores field. NOTE: Set the barChartMetric setting to 'mean' if you use this option of expMatrixToBarchartBed. For more information about expMatrixToBarchartBed or bedJoinTabOffset, run the program with no arguments to get a usage message. Example #4 In this example, you will use the bed6+5 file created in Example 3 to create a bigBarChart file, allowing the data to be remotely accessed and exist within a track hub. The track settings for bigBarChart on a hub can be viewed here. 1. If not already completed, follow steps 1-4 from Example 3 above, or download the example bed6+5 file here 2. Download the fetchChromSizes and bedToBigBed programs from the utilities directory appropriate to your operating system. 3. Use fetchChromSizes to create a chrom.sizes file for the UCSC database you are working with (hg38 for these examples). Alternatively, you can download the chrom.sizes file for any assembly hosted at UCSC from our downloads page (click on "Full data set" for any assembly). For example, the hg38.chrom.sizes file for the hg38 database is located at http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.chrom.sizes. 4. Save the autoSql file barChartBed.as to your computer. 5. Run bedToBigBed to create the bigBarChart file: bedToBigBed -as=barChartBed.as -type=bed6+5 inputBed hg38.chrom.sizes output.bigBed 6. Move the newly constructed bigBarChart file to a web accessible http, https, or ftp location. 7. Construct a custom track line with a bigDataUrl parameter pointing to the newly created bigBarChart file. If the matrix and category files used to make the precursor barChart file are also moved to an http, https, or ftp location, we can point to them on the custom track line as well (all settings must be on the same line): track type=bigBarChart name="bigBarChart Example One" description="A bigBarChart file" barChartBars="adiposeSubcut breastMamTissue colonTransverse muscleSkeletal wholeBlood" barChartMetric=median barChartUnit=RPKM bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/barChart/hg38.gtexTranscripts.bb barChartMatrixUrl=http://genome.ucsc.edu/goldenPath/help/examples/barChart/exampleMatrix.txt barChartSampleUrl=http://genome.ucsc.edu/goldenPath/help/examples/barChart/exampleSampleData.txt visibility=pack 8. To fully take advantage of creating a bigBarChart file and the barChartMatrixUrl and barChartSampleUrl supporting files, create a Track Hub and use a stanza such as the following: track exampleBarChartTrack type bigBarChart visibility full shortLabel exBarChart longLabel Simple example bar chart track barChartBars adiposeSubcut breastMamTissue colonTransverse muscleSkeletal wholeBlood barChartColors #FF6600 #33CCCC #CC9955 #AAAAFF #FF00BB barChartLabel Tissues barChartMetric median barChartUnit RPKM bigDataUrl http://genome.ucsc.edu/goldenPath/help/examples/barChart/hg38.gtexTranscripts.bb barChartMatrixUrl http://genome.ucsc.edu/goldenPath/help/examples/barChart/exampleMatrix.txt barChartSampleUrl http://genome.ucsc.edu/goldenPath/help/examples/barChart/exampleSampleData.txt Please note, the fields in your barChartBars line must match the terms in your categories file (exampleBarChartSamples.txt) in order for the boxplot display to show up on the details page for tracks. Below is an example image indicating the benefit of using these files in a hub, note the "View all data points for..." link that allows extracting data from the matrix file (exampleBarChartMatrix.txt) specific for this named item. [Example boxPlot image] Example #5 To help Track Hub Developers adjust the display of tracks we add two settings barChartBarMinWidth and barChartBarMinPadding. The first sets the minimum pixel width of the bars in the chart to a number of pixels, for example barChartBarMinWidth 10. The second sets the minimum pixel width between bars to a number of pixels, for example barChartBarMinPadding 5. Here are two example tracks using these settings on the same source data that can be loaded by going to the My Data, Custom Tracks page and pasting the below text to see how the display differs. browser position chr14:95,081,796-95,436,280 track type=bigBarChart barChartBarMinPadding=5 name="ex barChartBarMinPadding" description="A bigBarChart file with barChartBarMinPadding" barChartBars="adiposeSubcut breastMamTissue colonTransverse muscleSkeletal wholeBlood" visibility=pack bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/barChart/hg38.gtexTranscripts.bb track type=bigBarChart barChartBarMinWidth=20 name="ex barChartBarMinWidth" description="A bigBarChart file with barChartBarMinWidth" barChartBars="adiposeSubcut breastMamTissue colonTransverse muscleSkeletal wholeBlood" visibility=pack bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/barChart/hg38.gtexTranscripts.bb [Example1 of barChartBarMinPadding and barChartBarMinWidth] Both tracks have the same data, however, in the bottom track the barChartBarMinWidth 20 setting triggers wider widths, and the top track has larger padding between bars from the setting barChartBarMinPadding 5. As described in the settings entries for barChartBarMinWidth and barChartBarMinPadding there is a dynamic calculation dependent on the current window size, the width of the item, and the number of bars for the item. So that when zooming in the appearance of the barCharts with these settings can be different, at different scales. For instance, in the first image, you can see how much impact barChartBarMinWidth has on the second track, as well as the barChartBarMinPadding in the top track. But as zoomed in, with the below image, the impact of both of these settings is less noticeable. [Example2 of barChartBarMinPadding and barChartBarMinWidth] Example #6 To help with the selection and exploration of large data sets the new settings barChartFacets, barChartStatsUrl, and barChartMerge were introduced where on the details page checkboxes enable slicing data down to smaller collections based on metadata. The setting barChartFacets <column1,column2,...columnN> turns on the faceted selection on the track details and configure page which is useful for selecting which bars out of a large number to display by clicking designated checkboxes. The setting barChartStatsUrl &lturl> associates a table in tab-separated values with the barChart, with one line per bar. And the setting barChartMerge on enables a merge button inside of the faceted selections. It is particularly useful when there are many bars and many facets to condense a related group, such as tissue source. Below is an example track using these settings on source data for a Tabula Sapiens single cell RNA data from many tissues track. This excerpt of settings from that track allows experimenting to see these settings in action, and to be loaded by going to the My Data, Custom Tracks page. track type=bigBarChart name="ex Tabula Sapiens" description="A bigBarChart using Tabula Sapiens data to illustrate new Details pages" visibility=pack barChartCategoryUrl=http://hgdownload.soe.ucsc.edu/gbdb/hg38/bbi/tabulaSapiens/bw_edit_tissue_cell_type.categories barChartFacets=tissue,cell_class,cell_type barChartStatsUrl=http://hgdownload.soe.ucsc.edu/gbdb/hg38/bbi/tabulaSapiens/bw_edit_tissue_cell_type.facets barChartMerge=on bigDataUrl=http://hgdownload.soe.ucsc.edu/gbdb/hg38/bbi/tabulaSapiens/tissue_cell_type.bb Once loaded, click into an item to see the details page, in this case for the gene ACE2 at the default position in hg38. On the details page, rather than a static bar chart image, there is a dynamic interactive selection screen with checkbox facets to narrow down the display. Adding barChartMerge on enables the display of the "merge" button, and barChartFacets tissue,cell_class,cell_type sources information in barChartStatsUrl ...tissue_cell_type.facets to enable the facet options. To interact with this example, click the first two "merge" buttons next to "tissue" and "cell_class." [Example1 of Facets on barCharts] With those two merged selections, then click on the "Macrophage" option to see just this one cell type selection. [Example2git l of Facets on barCharts] By then clicking the "unmerge" button next to "tissue" the single bar chart will expand with tissue clusters. [Example3 of Facets on barCharts] In these ways the new barChartFacets, barChartStatsUrl, and barChartMerge settings allow users to explore the barChart data on the individual details page more closely. One can use the facets to further select certain types and also click the columns (val/count/cluster) to arrange by numerical value or alphabetical name. Also, if you click the "Return to Genome Browser" link, you will see only these selection bars are displayed. [Example4 of Facets on barCharts] In this image after making the selections browsing ACE2 the "zoom out" button has been clicked to also view nearby genes where the expression of these tissue selections for the gene PIR is quite noticeably different. Example #7 In this example, we will be using command-line tools that were used to create the single-cell tracks available on hg38. For more in-depth examples of these tools, take a look at the following makedoc for a real-life example. The matrixClusterColumns command converts a single cell gene expression matrix to a cell-type gene expression matrix. It takes a cell-by-cell metadata matrix that refers to the same cells as a gene expression matrix and combines the gene expression values for all cells of a given type into a single value representing the cell type. It can also be used on other metadata fields to produce matrices that show mean or average gene expression levels for a donor, an organ, or any other metadata field or combination of fields. The following command uses the exprMatrix.tsv and meta.tsv files to create six files: prepMatrix.tsv, prepStats.tsv, TissueCompMatrix.tsv, TissueCompStats.tsv, SexMatrix.tsv, and SexStats.tsv. matrixClusterColumns exprMatrix.tsv meta.tsv \ prep prepMatrix.tsv prepStats.tsv \ "Tissue Composition" TissueCompMatrix.tsv TissueCompStats.tsv \ Sex SexMatrix.tsv SexStats.tsv Read 5 rows from meta.tsv matrix exprMatrix.tsv has 209126 fields 209126 total columns, 209121 unclustered, 0 misses 209126 total columns, 209121 unclustered, 0 misses 209126 total columns, 209121 unclustered, 0 misses . If you are not sure which GENCODE Genes version is best suited for your data, the gencodeVersionForGenes command takes a list of gene symbols or gene accessions and searches for the version of GENCODE or RefSeq that matches the most genes in the list. Optionally, the tool can produce a BED file containing the gene structures for the genes in the list. The following command uses the gene.lst and geneSymVerTx.tsv files to create a mapping.bed file that will be used in the next step. # Figure out gene set gencodeVersionForGenes -target=hg38 gene.lst geneSymVerTx.tsv -bed=mapping.bed examining 23 versions of gencode and refseq best is gencodeVM5 as sym on mm10 with 6 of 6 (100%) hits on hg38 6 of 6 (100%) hit across versions The matrixToBarChartBed combines an expression matrix and a BED file with gene structures (mapping.bed) to make a new BED file (myTissueComp.bed) with a barChart showing gene expression that can be viewed on the Genome Browser. The optional argument, stats=stats.tsv provides a statistics file and improves the coloring in trackDb. The following command uses the TissueCompMatrix.tsv, mapping.bed, and TissueCompStats.tsv to create the myTissueComp.bed file that can be viewed on the Genome Browser or converted into a bigBed file so it can be used inside of a track hub. matrixToBarChartBed TissueCompMatrix.tsv mapping.bed myTissueComp.bed -stats=TissueCompStats.tsv 5 genes found, 0 (0.00%) missed The simpleBarChartBed.as can be used with the bedToBigBed command to create the bigBarChart file. bedToBigBed myTissueComp.bed hg38.chrom.sizes myTissueComp.bb -type=bed6+3 -as=simpleBarChartBed.as pass1 - making usageList (1 chroms): 15 millis pass2 - checking and writing primary data (5 records, 9 fields): 1 millis Sharing your data with others If you would like to share your barChart/bigBarChart data track with a colleague, learn how to create a URL by looking at Example 6 on this page. Extracting data from the bigBarChart format Because bigBarChart files are an extension of bigBed files, which are indexed binary files, it can be difficult to extract data from them. UCSC has developed the following programs to assist in working with bigBed formats, available from the binary utilities directory. - bigBedToBed — converts a bigBed file to ASCII BED format. - bigBedSummary — extracts summary information from a bigBed file. - bigBedInfo — prints out information about a bigBed file. Use the -as option to see the file field descriptions. As with all UCSC Genome Browser programs, simply type the program name (with no parameters) at the command line to view the usage statement. Troubleshooting If you encounter an error when you run the bedToBigBed program, check your input file for data coordinates that extend past the end of the chromosome. If these are present, run the bedClip program (available here) to remove the problematic row(s) in your input file before running the bedToBigBed program. 
/goldenPath/help/axt.html:Genome_Browser_axt_Alignment_Format	axt Alignment Format axt alignment files are produced from Blastz, an alignment tool available from Webb Miller's lab at Penn State University. The axtNet and axtChain alignments are produced by processing the alignment files with additional utilities written by Jim Kent at UCSC. Example: The following segment from an axt file shows the first 2 sets of alignments of the human assembly (the aligning assembly) to mouse chromosome 19 (the primary assembly). 0 chr19 3001012 3001075 chr11 70568380 70568443 - 3500 TCAGCTCATAAATCACCTCCTGCCACAAGCCTGGCCTGGTCCCAGGAGAGTGTCCAGGCTCAGA TCTGTTCATAAACCACCTGCCATGACAAGCCTGGCCTGTTCCCAAGACAATGTCCAGGCTCAGA chr19 3008279 3008357 chr11 70573976 70574054 - 3900 CACAATCTTCACATTGAGATCCTGAGTTGCTGATCAGAATGGAAGGCTGAGCTAAGATGAGCGACGAGGCAATGTCACA CACAGTCTTCACATTGAGGTACCAAGTTGTGGATCAGAATGGAAAGCTAGGCTATGATGAGGGACAGTGCGCTGTCACA Structure Each alignment block in an axt file contains three lines: a summary line and 2 sequence lines. Blocks are separated from one another by blank lines. Summary line 0 chr19 3001012 3001075 chr11 70568380 70568443 - 3500 The summary line contains chromosomal position and size information about the alignment. It consists of 9 required fields: - Alignment number -- The alignment numbering starts with 0 and increments by 1, i.e. the first alignment in a file is numbered 0, the next 1, etc. - Chromosome (primary organism) - Alignment start (primary organism) -- The first base is numbered 1 - Alignment end (primary organism) -- The end base is included. - Chromosome (aligning organism) - Alignment start (aligning organism) - Alignment end (aligning organism) - Strand (aligning organism) -- If the strand value is "-", the values of the aligning organism's start and end fields are relative to the reverse-complemented coordinates of its chromosome. - Blastz score -- Different blastz scoring matrices are used for different organisms.See the README.txt file in the alignments directory for scoring information specific to a pair of alignments. Sequence lines TCAGCTCATAAATCACCTCCTGCCACAAGCCTGGCCTGGTCCCAGGAGAGTGTCCAGGCTCAGA TCTGTTCATAAACCACCTGCCATGACAAGCCTGGCCTGTTCCCAAGACAATGTCCAGGCTCAGA The sequence lines contain the sequence of the primary assembly (line 2) and aligning assembly (line 3) with inserts. Repeats are indicated by lower-case letters. 
/goldenPath/help/hgTrackHubHelp.html:Track_Hubs	Using UCSC Genome Browser Track Hubs Contents What Are Track Hubs? What Are Assembly Hubs? Viewing Track Hubs Sharing Track Hubs Setting up a Track Hub Adding Groups to a Track hub Debugging Track Hubs Setting up item search Adding filters to your Track Hub Registering a Track Hub with UCSC Checking Hub settings and compatibility Where to host your data Track Hub Database Definition Document ------------------------------------------------------------------------ Additional resources - Raney BJ, et al. Track Data Hubs enable visualization of user-defined genome-wide annotations on the UCSC Genome Browser. Bioinformatics. 2014 Apr 1;30(7):1003-5. - Assembly Hubs Wiki - Track Database Definition Document (Reference guide for trackDb parameter definitions) - Track hub settings blog post - Public Hub Guidelines - Quick Start Guide to Basic Hubs - Quick Start Guide to Organizing Track Hubs into Groupings - Quick Start Guide to Assembly Hubs with Blat - Quick Start Guide to Searchable Track Hubs - Quick Start Guide to adding Filters to Hubs Search the Genome Browser help pages:   Questions and feedback are welcome. What Are Track Hubs? Track hubs are web-accessible directories of genomic data that can be viewed on the UCSC Genome Browser (please note that hosting hub files on HTTP tends to work even better than FTP and local hubs can be displayed on GBiB). Track hubs can be displayed on genomes that UCSC directly supports, or on your own sequence. Hubs are a useful tool for visualizing a large number of genome-wide data sets. For example, a project that has produced several wiggle plots of data can use the hub utility to organize the tracks into composite and super-tracks, making it possible to show the data for a large collection of tissues and experimental conditions in a visually elegant way, similar to how the ENCODE native data tracks are displayed in the browser. The track hub utility allows efficient access to data sets from around the world through the familiar Genome Browser interface. Browser users can display tracks from any public track hub that has been registered with UCSC. Additionally, users can import data from unlisted hubs or can set up, display, and share their own track hubs. Genome assemblies that UCSC does not support can be loaded and viewed with associated data. The data underlying the tracks and optional sequence in a hub reside on the remote server of the data provider rather than at UCSC. Genomic annotations are stored in compressed binary indexed files in bigBed, bigBarChart, bigGenePred, bigNarrowPeak, bigPsl, bigChain, bigInteract, bigMaf, bigWig, BAM, CRAM, HAL, hic or VCF format that contain the data at several resolutions. In the case of assemblies that UCSC does not support, genomic sequence is stored in the efficient twoBit format. When a hub track is displayed in the Genome Browser, only the relevant data needed to support the view of the current genomic region are transmitted rather than the entire file. The transmitted data are cached on the UCSC server to expedite future access. This on-demand transfer mechanism eliminates the need to transmit large data sets across the Internet, thereby minimizing upload time into the browser. The track hub utility offers a convenient way to view and share very large sets of data. Individuals wishing to display only a few small data sets may find it easier to use the Genome Browser custom track utility. As with hub tracks, custom tracks can be uploaded to the UCSC Genome Browser and viewed alongside the native annotation tracks. Custom tracks can be constructed from a wide range of data types; hub tracks are limited to compressed binary indexed formats that can be remotely hosted. However, the custom tracks utility does not offer the data persistence and track configurability provided by the track hub mechanism: hub tracks can be grouped into composite or super-tracks and configured to display the data using a wide variety of options. There is no way to create a browser on your own sequence with custom tracks. In general, for users who have large data sets that would be prohibitive to upload, need to ensure the persistence of their data, or would like to take full advantage of track functionality, or create a browser on sequence not natively supported by UCSC or a genome browser mirror, track hubs are a better solution. Both mechanisms give data providers the flexibility to directly add, update, and remove data from their display as needed. What Are Assembly Hubs? Assembly Data Hubs extend the functionality of Track Data Hubs to assemblies that are not hosted natively on the Browser. Assembly Data Hubs were developed to address the increasing need for researchers to annotate sequence for which UCSC does not provide an annotation database. They allow researchers to include the underlying reference sequence, as well as data tracks that annotate that sequence. Sequence is stored in the UCSC twoBit format, and the annotation tracks are stored in the same manner as Track Data Hubs. For more information on how to setup your own Assembly Data Hub, please refer to the Assembly Hub wiki and see the Quick Start Guide to Assembly Hubs. Viewing Track Hubs Public hubs The Genome Browser provides links to a collection of public track hubs that have been registered with UCSC. To view a list of the public track hubs available, click into the blue navigation bar "My Data" and then "Track Hubs" to reach the Public Track Hubs page. You can click links in the "Description" column to see details about a particular Hub. To view a hub's data, click on an assembly name on the row of your hub or the "Connect" button. If you clicked the "Connect" button, choose your assembly or click the "Genome Browser" link from the top blue bar to be brought to the default assembly. The selected hub tracks will be listed in a separate track group below the browser image and can be configured just like native browser tracks. Exercise caution when viewing a wide region that requires the Genome Browser to display a large number of track features: the browser display may time out. Unlisted hubs (located in the My Hubs tab) In addition to the Public Hubs listed, it is possible to load your own track hub or one created by a colleague. To add an unlisted hub, open the Track Hubs page and click the Connected Hubs tab. To import a new hub, type or paste its URL into the text box, then click the "Add Hub" button. If successful, your track hub will appear on that page. Tracks accessed through a hub can be used in Genome Browser sessions and custom tracks in the same manner as other tracks. The data underlying data hub tracks can be viewed, manipulated, and downloaded using the UCSC Table Browser. To remove a track hub from your Genome Browser display, click the "Disconnect" button on the Track Hubs page. For confidential or private data, please note that unlisted hubs are not secure. The URL helps to hide the location of the data; it is a simple barrier of obscurity. Please also know that hubs can be loaded from local directories when using GBiB. Sharing Track Hubs When sharing track hubs on a single assembly, we recommend using Saved Sessions (especially for publications). These have the advantages of being single-click access and sharing a full browser configuration. If you are just starting, an overview of the process is to make the hub on your web-accessible server, attach the hub to the Genome Browser, configure browser position and related tracks, then save your named session and share the session link. Your session links will be in the following format with your chosen username and session name: - http://genome.ucsc.edu/s/ExampleUser/TrackHubSession For additional information, see sharing Saved Sessions and backing up custom data. Creating a URL for a Track Hub Hubs can be loaded into the URL using the hubUrl= parameter. This parameter takes input similar to the track hub input box. Native UCSC supported genomes can be loaded into the URL using the db= parameter while non-natively supported genomes such as assembly hubs or GenArk hubs use the genome= parameter. URL parameters can be combined by using &. The following example links to the hg19 genome database and an example track hub using the db= and the hubUrl= parameters: http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg19&hubUrl=https://genome.ucsc.edu/goldenPath/help/examples/hubDirectory/hub.txt Track hubs' track visibility can also be changed from the URL parameters. As an example, the following link specifies: - the genome database (db=hg38) - loads a track hub (hubUrl=http://hgdownload.soe.ucsc.edu/hubs/gtex/hub.txt) - hides all tracks (hideTracks=1) - hides the subtrack kids of a particular track (gtexRnaSignalMaleYoung_hideKids=1) - sets a specific subtrack to be displayed (gtexRnaSignalSRR1311243=full) - ignores user settings (ignoreCookie=1). https://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&hubUrl=http://hgdownload.soe.ucsc.edu/hubs/gtex/hub.txt&hideTracks=1&gtexRnaSignalMaleYoung_hideKids=1&gtexRnaSignalMaleYoung=full&gtexRnaSignalSRR1311243=full&ignoreCookie=1 Optional parameters that can be added to the URL: - guidelines=on/off - activate or deactivate the blue guidelines - hgFind.matches=<listOfNames> - highlight features given their names - hgt.reset=1 - show only the default tracks - hgt.toggleRevCmplDisp=1 - show the reverse-complement - hgt.labelWidth=<number> - set the size of the left-side label area - hideTracks=1 - hide all tracks - hideTracks=1&<trackName>=full|dense|pack|hide - hide all tracks and show other tracks - highlight=<db>.<chrom>:<chromStart>-<chromEnd>#<color>|... - ignoreCookie=1 - do not load the user's existing settings saved in the internet browser's UCSC Genome Browser cookie. This means that the link will show the Genome Browser default settings such as track selections, custom tracks, and track hubs. Any changes you make in this new session will, however, affect the user's settings. E.g., if you add a track in this new window, and come back to the genome browser later, the track will still be there. This setting is useful if a website wants to link to the Genome Browser, starting with a "clean slate" but believes the user will come back to the Genome Browser expecting the changes to still be there. - ruler=hide - hide the ruler at the top of the browser image - oligoMatch=pack&hgt.oligoMatch=<dnaSeq> - switch on the Short Match track and highlight a matching sequence - pix=<number> - set the width of the image in pixels - textSize=<number> - set the size of text font - <trackName>=full|pack|dense|hide - show your current tracks, adding a track and set it to full, pack or dense visibility or hide it - <trackName>_imgOrd=<number> - vertically orders the tracks on the image based on the numbers provided. You need to specify an order for every visible track when using this parameter - <trackName>.heightPer=<number> - sets the height of the a bigWig track in pixels - <trackName>_hideKids=1 - hides a specific super track's individual tracks - <trackName>_sel=1 - selects specific subtrack to be 'checked', allowing display Creating a URL with Multiple Track Hubs You can create a URL with multiple Track Hubs using the hubUrl= parameter. URL parameters can be combined by using &. For example, using the following track hubs: ENCODE DNA Trackhub https://storage.googleapis.com/gcp.wenglab.org/hubs/dna20/hub.txt JASPAR TFBS http://expdata.cmmt.ubc.ca/JASPAR/UCSC_tracks/hub.txt ReMap 2022 Regulatory Atlas https://remap.univ-amu.fr/storage/public/hubReMap2022/hub.txt The combination of the three hubs allows the creation of the following URL that can load the three hubs on the Genome Browser. https://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&hubUrl=https://storage.googleapis.com/gcp.wenglab.org/hubs/dna20/hub.txt&hubUrl=http://expdata.cmmt.ubc.ca/JASPAR/UCSC_tracks/hub.txt&hubUrl=https://remap.univ-amu.fr/storage/public/hubReMap2022/hub.txt Creating a URL for an Assembly Hub The following example links to an assembly hub using the hubUrl= and genome= parameters where in the example, genome=araTha1, is the assembly name set for genome in the genomes.txt file. URL parameters can be combined by using &. https://genome.ucsc.edu/cgi-bin/hgTracks?genome=araTha1&hubUrl=http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubAssembly/plantAraTha1/hub.txt Additional hub connection parameters - hubClear=connects the hub and disconnects other hubs that are in the same directory - hgHubConnectReplacing hgTracks with this parameter connects the hub and redirects the link to the Track Data Hubs page - hgHub_do_firstDb=1 uses the first database in genomes.txt - hgHub_do_redirect=on redirects the attached hub to the Gateway page Redirecting the URL to the Gateway page The following example link connects the hub and redirects the link to the Gateway page to display the hub's description html page, which is defined in the genomes.txt by the htmlPath setting, by using hgHubConnect, hgHub_do_redirect=on, hgHubConnect.remakeTrackHub=on, hgHub_do_firstDb=1, and hubUrl. http://genome.ucsc.edu/cgi-bin/hgHubConnect?hgHub_do_redirect=on&hgHubConnect.remakeTrackHub=on&hgHub_do_firstDb=1&hubUrl=http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubAssembly/plantAraTha1/hub.txt You can also link to the Gateway page to display the hub's description html page by using the hgGateway and genome parameters. The following example links the hub to the Gateway page: http://genome.ucsc.edu/cgi-bin/hgGateway?genome=araTha1&hubUrl=http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubAssembly/plantAraTha1/hub.txt Creating a URL for an Assembly Hub with Multiple Track Hubs You can create a URL for an assembly hub with track hubs using a combination of the hubUrl= and genome= URL parameters. URL parameters can be combined by using &.For example, using the following assembly hub and track hubs: Arabidopsis thaliana assembly hub https://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubPlants/cshl2013/hub.txt ReMap 2022 Regulatory Atlas https://remap.univ-amu.fr/storage/public/hubReMap2022/hub.txt UniBind 2021 Robust hub https://unibind.uio.no/static/data/latest/UniBind_hubs_Robust/UCSC/hub.txt The combination of the three hubs, along with genome=araTha1, allows for the creation of the following URL that can load the three hubs on the Genome Browser. https://genome.ucsc.edu/cgi-bin/hgTracks?genome=araTha1&hubUrl=https://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubPlants/cshl2013/hub.txt&hubUrl=https://remap.univ-amu.fr/storage/public/hubReMap2022/hub.txt&hubUrl=https://unibind.uio.no/static/data/latest/UniBind_hubs_Robust/UCSC/hub.txt Creating a URL for a GenArk assembly with a track hub When creating a track hub for a GenArk assembly, there is no need to do any attaching of the assembly hub itself via the hubUrl= URL parameter. GenArk hubs will automatically attach themselves if the track hub mentions the GCA_ or GCF_ name identifier of the assembly hub. Simply load the track hub on the Genome Browser, and the assembly hub will automatically appear. For example, the following example track hub will load an additional track for the pig (GCA_002844635.1) GenArk assembly. To create a link to the track hub that references a GenArk assembly, the genome=GCA_002844635.1 and hubUrl= URL parameters can be used like in the following example: https://genome.ucsc.edu/cgi-bin/hgTracks?genome=GCA_002844635.1&hubUrl=https://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubGenArkExample/genArkTrackHub/hub.txt Setting up your own Track Hub This section provides a step-by-step description of the process used to set up a track hub on your own server. If you would like information about how to attach a track hub to an existing assembly hub, please refer to the following FAQ entry. To create your own hub you will need: - one or more data sets formatted in one of the compressed binary index formats supported by the Genome Browser: bigBed, bigBarChart, bigGenePred, bigNarrowPeak, bigPsl, bigChain, bigInteract, bigMaf, bigWig, BAM, CRAM, HAL, hic or VCF - a set of text files that specify properties for the track hub and for each of the data tracks within it - a twoBit file with your sequence if you are setting up an assembly hub. Note that the allowed characters in sequence names are [A-Za-z_0-9_-] - an Internet-enabled web server or ftp server The files are placed on the server in a file hierarchy like the one shown in Example 1. Users experienced in setting up Genome Browser mirrors that contain their own data will find that setting up a track hub is similar, but is usually much easier. Depending on the number and complexity of the data sets, a track hub can typically be set up in a day or two. It is generally easiest to run the command-line data formatting programs in a Linux programming environment, although it's possible to manipulate smaller data sets using Mac OS-X as well. Note: there is now a useOneFile on hub setting that allows the hub properties to be specified in a single file. More information about this setting can be found on the Genome Browser User Guide. If you would like to add metadata to your track hub, the following metadata guide contains examples of how to include the information in your tracks. Example 1: Directory hierarchy for a hub containing DNase and RNAseq data for the hg18 and hg19 human genome assemblies. The hg18/ and hg19/ subdirectories contain the assembly-specific data files. myHub/ - directory containing track hub files hub.txt - a short description of hub properties genomes.txt - list of genome assemblies included in the hub data hg19/ - directory of data for the hg19 (GRCh37) human assembly trackDb.txt - display properties for tracks in this directory dnase.html - description text for a DNase track dnaseLiver.bigWig - wiggle plot of DNase in liver dnaseLiver.bigBed - regions of active DNase liverGenes.bigGenePred - gene annotations of genes over-expressed in liver tissue dnaseLung.bigWig - wiggle plot of DNase in lung dnaseLung.bigWig - regions of active DNase ... rnaSeq.html - description text for an RNAseq track rnaSeqLiver.bigWig - wiggle plot of RNAseq data in liver rnaSeqLiver.bigBed - intron/exon lists for liver rnaSeqLung.bigWig - wiggle plot of RNAseq data in lung rnaSeqLung.bigBed - intron/exon lists for lung hg18/ - directory of data for the hg18 (Build 36) human assembly trackDb.txt - display properties for tracks in this directory dnase.html - description text for a DNase track dnaseLiver.bigWig - wiggle plot of DNase data in liver dnaseLiver.bigBed - regions of active DNase dnaseLung.bigWig - wiggle plot of DNase data in lung dnaseLung.bigWig - regions of active DNase ... rnaSeq.html - description text for an RNAseq track rnaSeqLiver.bigWig - wiggle plot of RNAseq data in liver rnaSeqLiver.bigBed - intron/exon lists for liver rnaSeqLung.bigWig - wiggle plot of RNAseq data in lung rnaSeqLung.bigBed - intron/exon lists for lung ------------------------------------------------------------------------ Step 1. Format the data The data tracks provided by a hub must be formatted in one of the compressed binary index formats supported by the Genome Browser: bigWig, bigBed, bigGenePred, bigChain, bigNarrowPeak, bigBarChart, bigInteract, bigPsl, bigMaf, hic, BAM, CRAM, HAL or VCF. Many of these formats also support several different chromosome aliases (e.g. '1' or 'NC_000001.11' in place of 'chr1'). bigWig - The bigWig format is best for displaying continuous value plot data, such as read depths from short read sequencing projects or levels of conservation observed in a multiple-species alignment. A bigWig file contains a list of chromosome segments, each of which is associated with a floating point value. When graphed, the segments may appear as a big "wiggle". Although each bigWig file can contain only a single value for any given base, bigWig tracks are often combined into "container multiWig" or "compositeTrack on" tagged tracks. For information on creating and configuring bigWig tracks, see the bigWig Track Format help page. bigBed - BigBed files are binary indexed versions of Browser Extensible Data (BED) files. BED format is useful for associating a name and (optionally) a color and a score with one or more related regions on the same chromosome, such as all the exons of a gene. See the bigBed Track Format help page for information on creating and configuring bigBed tracks. bigGenePred - BigGenePred files are binary indexed versions of Browser Extensible Data (BED) files with an extra eight fields that are useful for describing gene predictions that are modeled after the fields in genePred files. BigGenePred format is useful for associating a name and (optionally) a color and a score with one or more related regions on the same chromosome, such as all the exons of a gene. See the bigGenePred Track Format help page for information on creating and configuring bigGenePred tracks. bigChain - BigChain files are binary indexed versions of chain files. BigChain format is useful for large pairwise alignment data sets. See the bigChain Track Format help page for more information on creating and configuring bigChain tracks. bigNarrowPeak - BigNarrowPeak files are binary indexed versions of Browser Extensible Data (BED) files with first six fields being the same as bed, and an extra four fields that contain various scores and the offset of the base within the block that is the peak. See the bigNarrowPeak Track Format help page for information on creating and configuring bigNarrowPeak tracks. bigBarChart - BigBarChart files are binary indexed versions of barChart files. BigBarChart format is useful for bringing barChart display into track hubs, and supports schema customization and label configuration that is not supported for regular barChart format. See the barChart Track Format help page for information on creating and configuring bigBarChart tracks. bigInteract - BigInteract files are binary indexed versions of interact files. BigInteract format is useful for bringing interact display into track hubs, and supports schema customization and label configuration that is not supported for regular interact format. See the interact Track Format help page for information on creating and configuring bigInteract tracks. bigPsl - BigPsl files are binary indexed versions of PSL files. BigPsl format is useful for large data sets created by BLAT or other tools. See the bigPsl Track Format help page for more information on creating and configuring bigPsl tracks. bigMaf - BigMaf files are binary indexed versions of MAF files. BigMaf format is useful for large multiple alignment data sets. See the bigMaf Track Format help page for more information on creating and configuring bigMaf tracks. hic - Hic files are binary files that store contact matrices from chromatin conformation experiments. This format is useful for displaying interactions at a scale and depth that exceeds what can be easily visualized with the interact and bigInteract formats. See the hic Track Format help page for more information on creating and configuring hic tracks. BAM - BAM files contain alignments of (generally short) DNA reads to a reference sequence, usually a complete genome. BAM files are binary versions of Sequence Alignment/Map (SAM) format files. Unlike bigWig and bigBed formats, the index for a BAM file is in a separate file, which the track hub expects to be in the same directory with the same root name as the BAM file with the addition of a .bai suffix. See the BAM Track Format help page for more information. CRAM - The CRAM file format is a more dense form of BAM files with the benefit of saving much disk space. While BAM files contain all sequence data within a file, CRAM files are smaller by taking advantage of an additional external "reference sequence" file. This file is needed to both compress and decompress the read information. See the CRAM Track Format help page for more information. HAL - HAL (Hierarchical Alignment Format) is a graph-based structure to efficiently store and index multiple genome alignments and ancestral reconstructions. HAL files are represented in HDF5 format, an open standard for storing and indexing large, compressed scientific data sets. HAL is the native output format of the Progressive Cactus alignment pipeline, and is included in the Progressive Cactus installation package. VCF - VCF (Variant Call Format) files can contain annotations of single nucleotide variants, insertions/deletions, copy number variants, structural variants and other types of genomic variation. When a VCF file is compressed and indexed using tabix (available here), it can be used as a data track file. Unlike bigWig and bigBed formats, the tabix index is in a separate file, which the track hub expects to be in the same directory with the same root name as the VCF file with the addition of a .tbi suffix. See the VCF Track Format help page for more information. ------------------------------------------------------------------------ Step 2. Create the track hub directory Create a track hub directory in an Internet-accessible location on your web or ftp server. This directory will contain the hub.txt and genomes.txt files that define properties of the track hub and a subdirectory for each of the genome assemblies covered by the hub track data. Note: there is now a useOneFile on hub setting that allows the hub properties to be specified in a single file. More information about this setting can be found on the Genome Browser User Guide. ------------------------------------------------------------------------ Step 3. Place the track data files in an Internet-accessible location The data files underlying a track in a hub do not have to reside in the track hub directory or even on the same server, but they must be accessible via the Internet. The track hub utility supports Internet protocols such as http://, https://, and ftp://, as well as file paths relative to the hub directory hierarchy. The location of a track file is defined by its bigDataUrl tag in the associated trackDb.txt file (Step 7). ------------------------------------------------------------------------ Step 4. Create the hub.txt file Within the hub directory, create a hub.txt file containing a single stanza with up to six fields that define properties of the track hub: hub hub_name shortLabel hub_short_label longLabel hub_long_label genomesFile genomes_filelist email email_address descriptionUrl descriptionUrl hub - a single-word name of the directory containing the track hub files. Not displayed to hub users. This must be the first line in the hub.txt file. shortLabel - the short name for the track hub. Suggested maximum length is 17 characters. Displayed as the hub name on the Track Hubs page and the track group name on the browser tracks page. longLabel - a longer descriptive label for the track hub. Suggested maximum length is 80 characters. Displayed in the description field on the Track Hubs page. genomesFile - the relative path of the genomes.txt file, which contains the list of genome assemblies covered by the track data and the names of their associated configuration files. By convention the genomes.txt file is located in the same directory as the hub.txt file. email - the contact to whom questions regarding the track hub should be directed. descriptionUrl - URL to HTML page with a description of the hub's contents. This can be relative to the directory which holds hub.txt. This file is assumed to be HTML, and if the hub is a UCSC public hub, this HTML will be crawled nightly by UCSC to build an index with which public hubs can be searched. If present, clicks on the shortLabel will open this HTML in a new tab. This field is optional. useOneFile on - only use this setting if your hub is displaying one genome and if you wish to put all information into only one file: hub.txt. This setting allows you to put the text lines of genomes.txt and trackDb.txt all into the singular hub.txtfile. To learn more about this setting click useOneFile on and see a working example here. Example 2: Sample hub.txt file defining attributes for the track hub shown in Example 1. hub UCSCHub shortLabel UCSC Hub longLabel UCSC Genome Informatics Hub for human DNase and RNAseq data genomesFile genomes.txt email genome@soe.ucsc.edu descriptionUrl ucscHub.html ------------------------------------------------------------------------ Step 5. Create the genomes.txt file Create a genomes.txt file within the track hub directory that contains a two-line stanza that must be separated by a line for each genome assembly that is supported by the hub data. Each stanza shows the location of the trackDb file that defines display properties for each track in that assembly, as well as an optional metadata storage file genome assembly_database_1 trackDb assembly_1_path/trackDb.txt metaTab assembly_1_path/tabSeparatedFile.txt genome assembly_database_2 trackDb assembly_2_path/trackDb.txt metaDb assembly_2_path/tagStormFile.txt genome - a valid UCSC database name. Each stanza must begin with this tag and each stanza must be separated by an empty line. trackDb - the relative path of the trackDb file for the assembly designated by the genome tag. By convention, the trackDb file is located in a subdirectory of the hub directory. However, the trackDb tag may also specify a complete URL. metaDb - the path to an optional tagStorm file that has the metadata for each track. Each track with metadata should have a "meta" tag specified in the trackDb stanza for that track and a "meta" tag in the tagStorm file. metaTab - the path to an optional tab separated file that has the metadata for each track. Each track with metadata should have a "meta" tag specified in the trackDb stanza for that track and a "meta" tag in the tab separated file. The first line of the TSV file should start with a '#' and have the field names for each column, one of them being "meta". If this genomes.txt file is for an assembly that does not have native support in the browser, the following fields must also be present: twoBitPath - refers to the .2bit file containing the sequence for this assembly. Typically this file is constructed from the original fasta files for the sequence using the kent program faToTwoBit. See here for instructions on how to build a 2bit file. groups - a file which defines the track groups on this Genome Browser. Track groups are the sections of related tracks grouped together under the primary genome browser graphics display image. The groups.txt file defines the grouping of track controls under the primary Genome Browser image display. The example referenced here has the usual definitions as found in the UCSC Genome Browser. Each group is defined, for example the Mapping group: name map label Mapping priority 2 defaultIsClosed 0 The name is used in the trackDb.txt track definition group, to assign a particular track to this group. The label is displayed on the genome browser as the title of this group of track controls The priority orders this track group with the other track groups. The defaultIsClosed determines if this track group is expanded or closed by default. Values to use are 0 or 1. description - will be displayed for user information on the gateway page and most title pages of this genome assembly browser. It is the name displayed in the assembly pull-down menu on the browser gateway page. organism - the string which is displayed along with the description on most title pages in the Genome Browser. Adjust your names in organism and description until they are appropriate. This organism name is the name that appears in the genome pull-down menu on the browser gateway page. scientificName - specifies the scientific name for the assembly. defaultPos - specifies the default position the genome browser will open when a user first views this assembly. This is usually selected to highlight a popular gene or region of interest in the genome assembly. orderKey - used with other genome definitions at this hub to order the pull-down menu ordering the genome pull-down menu. htmlPath - refers to an html file that is used on the gateway page to display information for a novel assembly. Example genomes.txt including a newOrg1 assembly genome hg18 trackDb hg18/trackDb.txt genome hg19 trackDb hg19/trackDb.txt genome newOrg1 trackDb newOrg1/trackDb.txt twoBitPath newOrg1/newOrg1.2bit groups newOrg1/groups.txt description Big Foot V4 organism BigFoot defaultPos chr21:33031596-33033258 orderKey 4800 scientificName Biggus Footus htmlPath newOrg1/description.html ------------------------------------------------------------------------ Step 6. Create the genome assembly subdirectories Within the track hub directory, create a subdirectory for each of the genome assemblies that have track data in the hub. The subdirectory names must have a 1:1 correspondence with the database names defined by the genome tags in the genomes.txt file. ------------------------------------------------------------------------ Step 7. Create the trackDb.txt files The trackDb.txt file, which is based on the Genome Browser .ra format, is the most complicated of the text files in the hub directory. It contains a stanza for each of the data files for the given assembly that defines display and configuration properties for the track. If the tracks are grouped into larger entities, such as composite or super-tracks, the larger entities will have a stanza in the file as well. The Track Database Definition Document will help you understand how to create a trackDb.txt file. This document describes how to declare dataset display settings and values, and indicates the support level for each setting. While there are over 100 track settings supported at UCSC, other sites that display hubs have more limited settings support. To further portability of hubs, we have used input from other sites to identify a "base" subset of the "full" settings list, and the document has been assigned a version number. See the document introduction for a fuller explanation. At a minimum, each track in the trackDb.txt file must contain the "required" settings: track track_name bigDataUrl track_data_URL shortLabel short_label longLabel long_label type track_type track - the symbolic name of the track. The first character must be a letter, and the remaining characters must be letters, numbers, or under-bar ("_"). Each track must have a unique name. This tag pair must be the first entry in the trackDb.txt file. bigDataUrl - the file name, path, or Web location of the track's data file. The bigDataUrl can be a full URL. If it is not prefaced by a protocol, such as http://, https:// or ftp://, then it is considered to be a path relative to the trackDb.txt file. shortLabel - the short name for the track displayed in the track list, in the configuration and track settings, and on the details pages. Suggested maximum length is 17 characters. longLabel - the longer description label for the track that is displayed in the configuration and track settings, and on the details pages. Suggested maximum length is 80 characters. type - the format of the file specified by bigDataUrl. Must be either bigWig, bigBed, bigBarChart, bigGenePred, bigInteract, bigNarrowPeak, bigChain, bigPsl, bigMaf, hic, bam, halSnake or vcfTabix (Note: use type bam for CRAM files). If the type is bigBed, it may be followed by an optional number denoting the number of fields in the bigBed file (e.g., "type bigBed 12" for a file with 12 fields or "type bigBed 12 +" for a file that contains additional non-standard columns). If no number is given, a default value of 3 is assumed (a very limited display that omits names, strand information, and exon boundaries). Example 4: Sample trackDb.txt file containing two simple tracks. track dnaseSignal bigDataUrl dnaseSignal.bigWig shortLabel DNAse Signal longLabel Depth of alignments of DNAse reads type bigWig track dnaseReads bigDataUrl dnaseReads.bam shortLabel DNAse Reads longLabel DNAse reads mapped with MAQ type bam Suggestions: Default subtracks for composite For each composite, it is recommended that a subset of subtracks are "selected" (on) by default. This way, when a user turns the composite from hide to another visibility, they will see tracks displayed in the browser. Default composites within a super-track: For super-tracks that you don't want displaying by default when your track hub is turned on, it is recommended that some (or all) composites within the super-track be set to dense (or some visibility other than hide) by default and that the super-track be set to hide by default. This way, if a user changes a super-track from hide to show from the controls under the browser image, tracks are displayed. To implement, change the visibility line in trackDb of the super-tracks to hide and the visibility lines of all or some of the composite tracks within to dense (or some visibility other than hide). hgTrackUi controls: In addition to the controls for each view (click on the title of the view drop-down), there is often another set of controls above the view drop-downs (just under the "Overall display mode"). This set of controls is not associated with a particular view and clashes with the view controls. It is recommended to remove the controls that are not associated with a particular view. ------------------------------------------------------------------------ Step 8. Create track description files Each track in the hub may have an associated description file that describes the track to viewers. The file provides detailed information about the data displayed in the track, including methods used to produce and validate the data, background information, display conventions, acknowledgments, and reference publications. The description file, which must be in HTML format, is inserted into the track configuration page that displays when the user clicks on the track's short label. It also displays on the track details page that is shown when the user clicks on a feature in the track image. The track description file must have the same name as the symbolic name for the track (defined by the track tag in the trackDb.txt file) with a suffix of .html. For instance, a description file associated with the track named "dnaseSignal" in Example 4 would be named "dnaseSignal.html". The description file must reside in the same directory as the trackDb.txt file. Both parent and child tracks within a super-track can have their own description files. If the description file is not present, the corresponding sections of the track settings and details pages are left blank. Only one description page can be associated with composite and multiWig tracks; the file name should correspond to the symbolic name of the top-level track in the composite. Adding groups to a Track Hub To organize tracks within your track hub, you can utilize the groups setting, which references a separate text file that defines track groups for display under the genome browser graphic. [] The groups setting can be applied to a UCSC genome, a GenArk assembly, or an assembly hub. These track hub groups are kept separate from other track hubs and the native UCSC Genome Browser track groups, allowing for greater organizational flexibility. For instance, you can add a "genes" group without causing conflicts or confusion. You can define groups with names like "Category 1", "Category 2", and "Category 3". These group names are specified in the "groups.txt file ", which defines the track groups. Below is an example stanza demonstrating how to define these groups: name category1 label Category 1 name category2 label Category 2 priority 1 name category3 label Category 3 defaultIsClosed 1 You can access the complete "groups.txt " file containing these group definitions here: groups.txt. Within this file: - The name setting is used in the trackDb.txt file to associate specific tracks with a group. - The label setting specifies the title of the group in the genome browser. By default, groups are sorted alphabetically based on the label. - The priority setting dictates the display order of the track groups, with lower numbers shown first. - The defaultIsClosed setting controls whether the group is initially expanded or collapsed (0 for expanded, 1 for collapsed). After defining the "groups.txt" file, tracks can be assigned to a specific group within the "trackDb.txt" file using the group setting. For example, to assign the "bigBed1" track to the "category1" group, you could use the following stanza: track bigBed1 bigDataUrl https://hgdownload.soe.ucsc.edu/gbdb/hg38/ncbiRefSeq/ncbiRefSeqGenomicDiff.bb shortLabel bigBed example 1 longLabel This bigBed file is an example from the NCBI RefSeq Diffs Track type bigBed group category1 visibility dense The full "trackDb.txt" file, which includes the "bigBed1" track and additional examples, can be viewed here: trackDb.txt. Next, incorporate the "groups.txt" file with the groups setting into the genome stanza. This can be for a UCSC genome, a GenArk assembly, or an assembly hub. Below is an example using the hg38 genome: genome hg38 trackDb hg38/trackDb.txt groups groups.txt An example "genomes.txt" file, which includes the hg38 genome, a GenArk assembly, and an assembly hub, is available here: genomes.txt. By attaching the hub with these group settings, the tracks will be displayed within the specified groups. Below are URLs showing the group settings for the hg38 genome, a GenArk assembly, and an assembly hub: - hg38: hg38 Genome. - GCF_000951035.1: GenArk Assembly. - Arabidopsis thaliana Assembly Hub: Assembly Hub. Debugging Track Hubs Not updating? Change udcTimeout As part of the track hub mechanism, UCSC caches data from the hub on the local server. The hub utility periodically checks the time stamps on the hub files, and downloads them again only if they have a time stamp newer than the UCSC one. For performance reasons, UCSC checks the time stamps every 300 seconds, which can result in a 5-minute delay between the time a hub file is updated and the change appears on the Genome Browser. Hub providers can work around this delay by inserting the CGI variable udcTimeout=5 into the Genome Browser URL, which will reduce the delay to five seconds. To add this variable, open the Genome Browser tracks page and zoom or scroll the image to display a full browser URL in which the CGI variables visible. Insert the CGI variable just after the "hgTracks" portion of the URL so that it reads http://genome.ucsc.edu/cgi-bin/hgTracks?udcTimeout=5& (with the remainder of the URL following the ampersand). To restore the default timeout, a warning message will appear on hgTracks with a link to clear the udcTimeout variable. I used udcTimeout, why is it still not updating? Browser software attempts to speed performance by reading the trackDb architecture into a separate cache. This software assumes that the timestamp on a file like a hub's trackDb.txt should always be increasing in time and never decreasing (i.e., a new update to the file should not have an earlier timestamp). We did discover that if someone were to copy an older file (say with cp -p to preserve file timestamp), for example to restore a trackDb.txt to an earlier version, they might get stuck with a hub that is not updating. To resolve this specific problem the person would only need to edit or touch the file and give it a new timestamp. Check hub settings using Hub Development tool Hubs can be checked for valid file configuration, trackDb keywords, and composite or super track settings with the Hub Development tool. This tool can be accessed from the Track Hub page under the Hub Development tab. After entering your URL in the search box, the tool runs the hubCheck utility program equivalent to: hubCheck -noTracks http://url.to.hub.txt The noTracks setting speeds up the validation process but does not check for the presence or validity of your remotely hosted data track files. This tool checks your hub.txt, genome.txt, and trackDb.txt settings and displays warnings and errors in bright red font, such as "Missing required setting..." and "Cannot open...". The "Display load times" and "Enable hub refresh" optional settings show the load timing at the bottom of the Genome Browser page and allow instant hub refresh instead of 5 minute refresh. These options can be checked and activated by clicking "View Hub on Genome Browser". The following picture shows the example track grouping hub with the warning that the hub has no hub description page, no configuration errors, and "Display load times" checked: [The Hub Development tool checks config setting] The Hub Development tool checks for proper configuration files and track hub settings, and allows access to debugging settings. Check hub settings using hubCheck utility It is a good practice to run the command-line utility hubCheck on your track hub when you first bring it online and whenever you make significant changes. This utility by default checks that the files in the hub are correctly formatted, but it can also be configured to check a few other things including that various trackDb settings are correctly spelled and that they are supported by the UCSC Genome Browser. You can read more about using hubCheck to check the compatibility of your hub with other genome browsers below. Here is the usage statement for the hubCheck utility: hubCheck - Check a track data hub for integrity. usage: hubCheck http://yourHost/yourDir/hub.txt options: -checkSettings - check trackDb settings to spec -version=[v?|url] - version to validate settings against (defaults to version in hub.txt, or current standard) -extra=[file|url] - accept settings in this file (or url) -level=base|required - reject settings below this support level -settings - just list settings with support level Will create this directory if not existing -noTracks - don't check remote files for tracks, just trackDb (faster) -udcDir=/dir/to/cache - place to put cache for remote bigBeds and bigWigs Note that you will have to use the -udcDir option if /tmp/udcCache is not writable on your machine. The hubCheck program is available from the UCSC downloads server at http://hgdownload.soe.ucsc.edu/admin/exe/. Troubleshooting Track Hub connections If the browser is unable to load a track hub, it will display an error message. Some common causes for an import to fail include typos in the URL, a hub server that is offline, or errors in the track hub configuration files. Occasionally, remote track hubs may be missing, off-line, or otherwise unavailable. If a user is already browsing data from the remote hub when it disconnects, a yellow error message will be displayed instead of the expected data. Download all files in a hub using hubClone utility You can use the command-line utility hubClone to download all of the data and configuration files for a hub onto your local machine. To do so, use the utility with the "-download" option and it will download any files specified by bigDataUrls (e.g. bigBed, bigWig, etc). By default, it only copies the configuration files and changes any local bigDataUrls into remote URLs. Beyond that, you may also find it useful to make a copy of a public hub's configuration files in order to imitate various settings, such as filters, search options, or track groupings. Here is the usage message for the hubClone utility: hubClone - Clone the remote hub text files to a local copy in newDirectoryName, fixing up bigDataUrls to remote location if necessary usage: hubClone http://url/to/hub.txt options: -udcDir=/dir/to/udcCache Path to udc directory -download Download data files in addition to the hub configuration files Note that you will have to use the -udcDir option if /tmp/udcCache is not writable on your machine. The hubClone program is available from the UCSC downloads server at http://hgdownload.soe.ucsc.edu/admin/exe/. Setting up track item search The Genome Browser supports searching for items within bigBed tracks in track data hubs. To support this behavior you have to add an index to the bigBed file when you initially create the the bigBed file from the bed file input. Indices are usually created on the name field of the bed, but can be created on any field of the bed. Free-text searches can also be enabled by creating a TRIX index file that maps id's in the track to free-text metadata. Further instructions can be found in the Searchable Hub Quick Start Guide. See the searchIndex and searchTrix fields in the Hub Track Database Definition document for information on how to set up your bigBed to enable searching. The searchIndex setting requires the input BED data to be case-senstive sorted (sort -k1,1 -k2,2n). You can use either the example UNIX sort command or the bedSort utility available here. See an example searchable hub. Adding Filters to your Track Hub The Genome Browser supports three varieties of data filter options for bigBed data. These can improve data usability in many ways, such as displaying specific data by default (e.g. only items that pass certain quality scores), allow for filtering based on pre-specified categories (e.g designate only LINE repetitive elements from a list of options) and more. For more information on the options available and examples on how to set up filters, see our Track Hub Filters Quick Start Guide and the filter entries in the Track Database Definition Document. Registering a Track Hub with UCSC If you would like to share your track hub with other Genome Browser users, you can register your hub with UCSC by contacting the Genome Browser technical support mailing list at genome@soe.ucsc.edu. Please include the URL of your hub.txt file in the message. Once registered, your hub will appear as a link on the Public Hubs tab on the Track Hubs page. To assist developers of Public Hubs, there is a Public Hub Guidelines page. The page shares pointers and preferred style approaches, such as the need for creating description html pages for your data that display any available references and an email contact for further data questions. Alternatively, you can share your track hub with selected colleagues by providing them with the URL needed to load your hub via the Connected Hubs tab. Checking Hub settings and Compatibility Due to the growth in popularity of the track hub format, other genome browsers have begun supporting the UCSC track hub format. The hubCheck utility can be used to check the compatibility of your hub with the UCSC Genome Browser and other genome browsers. The following examples describe various settings that may be useful to you as you test this compatibility. Example 1: Listing the trackDb settings and their support levels and filtering them by support level. You can see all of the currently supported trackDb settings and their support level in the UCSC Genome Browser by running hubCheck with the "-settings" option: $ hubCheck -settings You can filter the displayed settings by support level by using the "-level=" option followed by the maximum level you wanted displayed. For example, to show only the required settings: $ hubCheck -settings -level=required Or, to show settings at both the required and base support levels: $ hubCheck -settings -level=base When you use the "-level" option to declare a support level, it includes the level that you define plus every level above it. This hierarchy of the different support levels is defined at the beginning of the Track Database Definition document. Example 2: Checking your trackDb settings against the settings and support levels provided by the UCSC Genome Browser. The "-checkSettings" option can be used check the settings in your hub's trackDb file against those provided by the UCSC Genome Browser on the Track Database definition page. This does not check to see that all of these settings are properly used, but checks to see that they are supported by the Genome Browser. For example, to fully run hubCheck on your hub: $ hubCheck -checkSettings http://genome.ucsc.edu/goldenPath/help/examples/hubDirectory/hub.txt If you are just looking to check the compatibility of your hub and not the integrity of your files, you can use the "-noTracks" option to just check setting compatibility. Skipping these file integrity checks will also speed up hubCheck. Here is an example of some of the errors you might see if your hub includes unsupported settings: $ hubCheck -checkSettings http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubCheckUnsupportedSettings/hub.txt Found 1 problem: Setting 'ensemblAssemblyName' is unknown/unsupported If you want to check the settings used in your hub against those at a particular support level and higher, you can also use the "-level=" setting here as well. For example: $ hubCheck -checkSettings -level=base http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubGroupings/hub.txt The resulting list of problems reported by hubCheck are settings beyond the base support level and are less likely to be supported at other genome browsers, as not all external browsers support the full list of UCSC settings. We will periodically increment the version number for our trackDb settings as large numbers of settings are added, updated or (rarely) removed. Using just the "-checkSettings" option will check your trackDb settings against those defined in the most recent version of the Track Database definition page. However, if you want to check your settings against an older version of these settings, you can use the "-version=" option. For example, if you wanted to check your hub against version one of our trackDb settings: $ hubCheck -checkSettings -version=v1 http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubGroupings/hub.txt Example 3: Checking your settings against those provided by UCSC and another source, such as Ensembl. If you want to check the settings in your hub against those supported by other genome browsers, you will first need to create a single-column file that lists each non-UCSC setting and then use the "-extra=" option to specify this file when running hubCheck. For example, if you knew that a setting called "ensemblAssemblyName" was supported for use in track hubs by Ensembl, you could create a single line file that included the setting "ensemblAssemblyName". Then, when you want to check a hub that includes these extra trackDb settings, you would then specify this extra settings file on the command line: $ hubCheck -checkSettings -extra=http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubCheckUnsupportedSettings/myExtraSettings.txt http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubCheckUnsupportedSettings/hub.txt (Note: The settings listed here in the "extra" file are just examples and do not represent real trackDb variables for hubs at Ensembl.) Where to host your data? As stated in What Are Track Hubs?, track hubs files must be located in web-accessible locations that support byte-range requests. Four options for hosting include: - Your institution's Information Technology services - Commercial webspace providers - Commercial cloud providers - Free webspace providers Your Institution: Many universities provide a location for researchers to place shareable data on the web and contacting your institution's system administrators will help discover a location to store your data. For example, if you work at the NIH, there is an internal data sharing NIH network site. Sometimes institution firewall rules can change, and you may need to inform your system administrators to add browser IP addresses as exceptions, listed here. Usually, your IT department can direct you to someone who manages webspace for individual groups. This is our recommended option, as it is usually free, very fast and you can update the files yourself easily. Webspace providers: If your institution does not provide any web hosting space for you, the most convenient solution is usually to buy a virtualized webspace server from a commercial web hosting provider. Files can be uploaded with FTP, rsync or scp and appear on a https:// domain. Avoid unlimited offers, they often do not allow binary files and are slower, rather look for a "virtual private server" (VPS). Some examples of providers are: A2 Hosting, BlueHost, GoDaddy, HostGator, Hostinger, DreamHost, but there are many others. This is not a complete list and we do not endorse a particular one. You can search the internet for "virtual private server comparison". Offers start at around $5-10/month for 25-50 GB of storage. The advantage of VPS providers is that they bill a flat rate per month, which may be easier to order through universities than the per GB transferred billing of cloud providers. For optimal performance, select a West Coast / San Francisco data center when ordering a web server, as this is closest and fastest from UCSC. There are usually no backups included, so it is good to keep local copies of your files. Cloud providers: In general, commercial online cloud backup providers that charge a flat rate (Dropbox, iCloud, Google Drive, Box.com, Microsoft OneDrive, Tencent Weiyun, Yandex.Disk, etc.) do not work reliably as their business model requires rare and rate-limited data access, which is too slow or too limited for genome annotation display. However, commercial cloud storage offers that charge per GB transferred (Amazon S3, Microsoft Azure Storage, Google Cloud Storage, Backblaze, Alibaba Object Store, etc.) typically do work. As of 2020, they cost around 2-3 US cents/GB/month to store the hub data and 12-18 US cents per GB transferred, when the hub is used. For optimal performance, select a San Francisco / San Jose data center for the main UCSC site genome.ucsc.edu, a Frankfurt/Germany data center for genome-euro.ucsc.edu and a Tokyo data center for genome-asia.ucsc.edu. You may also want to review this discussion about issues with distributed storage servers. These services are external to UCSC and may change. Free webspace: If you do not want to pay for web space, and your institution does not provide a data location supporting byte-range requests, we know of at least the following sites where you can host research data and configuration files for free: - Galaxy - Maximum size limit is 50 GB (uncompressed). 250 GB of storage is available per Galaxy account. - CyVerse Discovery Environment - 5 GB of space for free, but can be relatively slow to display. Offers a paid subscription service to expand space. - Github - files limited to 100MB, but very fast. - Figshare - not limited and fast, but every file needs to be uploaded individually and cannot be changed. Optimal for very stable links, e.g. in publications. - DropBox - 2 GB free on their BasicFree plan. Many other paid dropbox plans are available with much more space. Each of the providers above has a slightly different approach to hosting data for compatibility with the UCSC Genome Browser, and may have different advantages and disadvantages, such as size limitations, usage statistics, and version control integration. Additionally, as previously mentioned, any provider that supports byte-range access will work for hub hosting, and you are not limited to the above sites. Below is a summarized guide for each of the providers mentioned above. Galaxy Hosting Cyverse Hosting GitHub Hosting Figshare Hosting DropBox Hosting Hosting Files on Galaxy Galaxy is an open-source platform for FAIR data analysis that enables users to streamline and the analysis of genomic data and serves as a comprehensive toolkit for researchers, scientists, and bioinformaticians. The Galaxy platform provides a user-friendly web interface that allows you to host your data. You can navigate to the Galaxy platform, log in to your account (or create an account if you haven't already), and use the data upload functionality provided to host your data. Once uploaded, the data will be stored securely on the platform and made available for your analysis. Viewing a Single File Hosted on Galaxy 1. To begin, sign into your Galaxy account and then click the "Upload Data" button on the left-hand side of the page. A pop-up will appear to upload the file to Galaxy. [] 2. On the pop-up menu, click the "Choose local files" to select the file to host. Once the file is selected, select the genome and file type from the drop-down menus, and click the "Start" button. For this example, we will upload a bigBed file for the GRCh37/hg19 genome to Galaxy. [] 3. Once the file is uploaded to Galaxy, it will appear in the "History" section on the right-hand side. Click on the file to expand the menu, and additional information about the file such as the file size, format, and the database. Towards the bottom of the menu, five icons are displayed that perform different functions on Galaxy. Clicking on the "graph icon" will bring up another menu in the middle of the screen with a link to visualize the bigBed file on the UCSC Genome Browser. [] Creating and Viewing a Hub Hosted on Galaxy 1. Similar to the step in the previous example, sign into your Galaxy account and upload the files to Galaxy. For this example, we will be uploading a bigBed, a VCF, and a VCF index file to view on the GRCh37/hg19 genome as a track hub. [] 2. To create the hub, a URL to each of the files is needed to reference the Galaxy hosted data inside the hub. Click on each of the three files to expand the menu and display the five icons at the bottom. Next, click on the "chain-link" icon to copy the URL to the hosted file. With these three URLs, we can now begin to build the hub. [] 3. When creating the hub, it is recommended to use the useOneFile on setting to have the contents of the genomes.txt, hub.txt, and trackDb.txt files inside of a single file. With the URLs from the previous step, reference the three files hosted on Galaxy using the the bigDataUrl and bigDataIndex trackDb settings. The entire contents of the example hub can be seen below: hub myExampleHub shortLabel Example Hub longLabel Example Hub Hosted on Galaxy useOneFile on email genome-www@soe.ucsc.edu genome hg19 track vcfExample shortLabel VCF example longLabel VCF example using Galaxy visibility pack type vcfTabix maxWindowToDraw 200000 bigDataUrl https://usegalaxy.org/api/datasets/f9cad7b01a472135076a05ca3c1069a7/display?to_ext=vcf bigDataIndex https://usegalaxy.org/api/datasets/f9cad7b01a47213595c3df01954cabaf/display?to_ext=binary track bigBedExample shortLabel BigBed example longLabel BigBed example using Galaxy type bigBed visibility pack bigDataUrl https://usegalaxy.org/api/datasets/f9cad7b01a4721358405ebd3fc34f65a/display?to_ext=bigbed 4. After creating the hub on your local computer, upload the hub.txt file to Galaxy. Once the file is on Galaxy, click on the hub.txt file to expand the menu. Again, click on the "chain-link" icon to obtain the URL to the hub.txt file. Now, navigate to the UCSC Genome Browser's Track Hubs page and enter the URL to view the hub. [] 5. Alternatively, you can create a URL using the &hubUrl= URL parameter to quickly share and load the hub with colleagues: http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg19&position=chr21:33031468-33034434&hubUrl=https://usegalaxy.org/api/datasets/f9cad7b01a472135a3056c5f93f137db/display?to_ext=txt Hosting Hubs on CyVerse CyVerse, previously known as the iPlant Collaborative, is an NSF-funded site created for assisting data scientists with their data storage and compute needs. Data hosting by CyVerse is free for academic groups and they support byte-range access, so they can be used for track hubs. However, Cyverse is sometimes slow, and may result in error messages if your hub includes many tracks that are meant to be shown at the same time by your users. In order to host your data on CyVerse, you first must create an account and then use their Discovery Environment to upload data. After creating an account and signing in, access the data screen by clicking the second icon on the left. Use the "Upload" button on the far right to import data from a URL or locally from your machine. [] You can also use the command line utility iCommands to facilitate bulk transfer of data (best used for large files in the 2-100 GB range), or use Cyberduck to bulk transfer up to 80 GB of data in one go. Once your file is available use the three dots on the far right to click the "Public Links(s)" option. [] Select this option for all the files you will be using with the Genome Browser, whether they are text-based files (trackDb.txt, groups.txt, description.html, etc.) or binary-indexed files (BAM, bigWig, bigBed, etc.) requiring byte-range access. Note, if you have a dataFile.bam, you must also have a dataFile.bam.bai file of the matching name and both must have public links created. After creating public links to your binary files, you must ensure your text files (i.e., trackDb.txt) point to the CyVerse locations for the files. For instance, the bigDataUrl setting, will need to point to the location of the BAM, bigWig, or bigBed (i.e., bigDataUrl https://data.cyverse.org/... /dataFile.bam). [] The hub.txt file (if not using the useOneFile on setting) will need to point the related genomes.txt location, which in turn points to the trackDb.txt location using these full https://data.cyverse.org/... links as well. Please see the Using the Discovery Environment wiki page on the CyVerse wiki for more information. Please direct any questions about CyVerse or the Discovery Environment to their Contact Us page or "Chat with Cyverse support" staff directly via the blue question box icon on the top right of the Discovery Environment page. [] Note, if you need to replace files once they have been uploaded into CyVerse's DE and Public links have already been created you will need to force update the CyVerse cache. One way is to go back to the Public links section and find "Refresh Cache" button. Another way is by hitting Control-Shift-R in your browser to force reload the file or by sending Cache-Control: no-cache header: curl --head --header 'Cache-Control: no-cache' https://data.cyverse.org/.../dataFile.bam Hosting Hubs on Github Github supports byte-range access to files when they are accessed via the raw.githubusercontent.com style URLs. To obtain a raw URL to a file already uploaded on Github, click on a file in your repository and click the Raw button: [Location of the Raw button for generating a plaintext URL to a file hosted on Github.] The "Raw" button results in a plain text page like the following: [Example of URL to a file on github] The bigDataUrl field (and any other statement pointing to a URL like bigDataIndex, refUrl, barChartMatrixUrl, etc.) of your trackDb.txt file should use the "raw.githubusercontent.com" style URL as shown above. Note that similar to CyVerse, you will need to update your trackDb.txt bigDataUrl lines to point to their correct raw.githubusercontent.com addresses. Please also note that HTML does not render properly from this address, and so you if you are also hosting your hub's description page here, you will want to use a site like RawGit to point to your descriptionUrl. The advantage to hosting your data on Github is the built in version control of the site, meaning you can view previous hub configurations and keep backups and history of your work automatically. The disadvantage of hosting on github is the relatively small file size upload limit compared to other hosting providers. For an example public hub hosted on Github, please see the Human cellular microRNAome barChart hub. For more information about moving files to Github, please see Github's help pages. Please direct any questions about Github to their help desk. Hosting hubs on figshare Figshare is a site for researchers and institutions to upload and collect usage statistics on their data, as well as make their data shareable and discoverable. The process for uploading a hub to Figshare is similar to the process involved at CyVerse, where one must first create an account, upload the bigDataUrl files, create shareable links, and then edit your hub.txt, genomes.txt, and trackDb.txt appropriately. One advantage to using Figshare is their emphasis on usage statistics, so institutional accounts can see how often their hubs and tracks are being accessed by others. Note that Figshare does not use filenames as part of the URLs, therefore bigDataUrl files that require a separate index file, like VCFs and BAM files, must have their index file location specified with a bigDataIndex. This keyword is relevant for Custom Tracks and Track Hubs. You can read more about bigDataIndex in the TrackDb Database Definition page. If you are having issues hosting at figshare, try to use the file's download URL. This URL will have "ndownloader" in its path. Also, for custom tracks you will need to declare a track line with track type and bigDataUrl. Below is a simple example of a bigBed custom track on hg38: track type=bigBed name="figshare example" bigDataUrl=https://figshare.com/ndownloader/files/38068053 Hosting hubs on DropBox DropBox is a site for users to share data files. DropBox has recently added support byteranges, a feature we need for hubs and custom tracks. DropBox has 2 GB in their BasicFree plan for free. Dropbox Plus for $10 a month -- $120 per year, and you must pay the full year to get the lowest price -- 1 user with 2 TB files up to 50GB maximum individual file size. They have lots of other plans available to for users, businesses, and schools. One must first create a DropBox account, upload the bigDataUrl files, or drag-and-drop from local disk. DropBox also has an app. They provide support for MacOS and Android too. Copy the share URL provided. Edit your hub.txt appropriately. Use useOneFile on hub setting that allows the hub properties to be specified in a single file. More information about this setting can be found on the Genome Browser User Guide. If you would like to add metadata to your track hub, the following metadata guide contains examples of how to include the information in your tracks. Click share on each data file, choose copy link, Ctrl-C and paste the dropbox URL into the hub oneFile the bigDataUrl field in the trackDb section of your useOneFile hub txt. It will look somehting like this: bigDataUrl https://www.dropbox.com/scl/fi/8t785o3sqidp0tmar91bf/dnaseRep3.bw?rlkey=37wucbhdvwqntw4ejvig4kg7c&st=11v8l216&dl=0 Click the share button over your hub .txt file and Copy Link, Ctrl-C. Paste that dropbox hub url into the hgHubConnect CGI tab. https://www.dropbox.com/scl/fi/6wrobg6wqcgtm7khew4qo/hub1.txt?rlkey=vi32q9tb68kjpn2xhy0qjuqkf&st=nldzp2tw&dl=0 Note that DropBox does not use pathnames as part of the URLs, therefore bigDataUrl files that require a separate index file, like VCFs and BAM files, must have their index file location specified with a bigDataIndex. This keyword is relevant for Custom Tracks and Track Hubs. You can read more about bigDataIndex in the TrackDb Database Definition page. For custom tracks you will need to declare a track line with track type and bigDataUrl. Below is a simple example of a bigBed custom track on hg38: track type=bigWig name="dropbox example" bigDataUrl=https://www.dropbox.com/scl/fi/8t785o3sqidp0tmar91bf/dnaseRep3.bw?rlkey=37wucbhdvwqntw4ejvig4kg7c&st=1z1jce0t&dl=0 For more information on using DropBox, please see their DropBox Support. DropBox also provides a folder feature to help organize and group related data. Troubleshooting your own HTTPS server configuration When your own institution's system administrators are hosting your data they may benefit from this section about ensuring a secure HTTPS configuration. The most popular web servers that system admins use are Apache and NGINX. Instructions for setting up these popular web servers are found all over the web, so this section will not cover those here. Certs and Security As security on the Internet becomes increasingly important, SSL certificates are often required for proper server installation. Proper certificate validation helps stop "Man-In-The-Middle" attacks by ensuring that connections go to the correct server and not some fake imposter site. This process requires SSL certificates that have not expired, and whose domain name matches the domain name specified in the HTTPS URL. The UCSC Genome Browser's networking software uses the very popular open source library openssl 1.0. System administrators hosting your data should ensure that TLS1.2 is allowed if you are going to provide data over HTTPS, since it is fast and secure and compatible with openssl 1.0. FREE CERT PROVIDER To help system administrators, here are groups that provide free web certs, including the popular LETSENCRYPT Testing your site certs Here are ways to check HTTPS certificates, such as with curl, which uses openssl. curl https://yourdomain.com/yourhub/hub.txt If curl can fetch the hub.txt HTTPS URL without errors, then the certs should work with the UCSC Genome Browser. For a deeper level of debugging, system administrators can use the open ssl client command: openssl s_client -trusted_first -connect yourdomain.com:443 -servername yourdomain.com Various online SSL Server Test sites have great detailed documentation about how to check your website's certs and configuration, such as https://www.ssllabs.com/ssltest/. Here is an example where you can supply yourdomain.com and discover results: https://www.ssllabs.com/ssltest/analyze.html?d=yourdomain.com&latest Feel free to contact UCSC Genome Browser for help if you are seeing certificate validation error messages you do not understand. 
/goldenPath/help/liftOver.html:Genome_Browser_Track_LiftOver	LiftOver of tracks from previous to new assembly The tracks indicated by a "ball" logo (e.g., [] and []) have been lifted from a previous assembly of the same organism with a minimum of quality control scrutiny (e.g., have been lifted from hg17 or hg18 to a later human assembly). The number indicated on the logo indicates the version of the assembly that the track was lifted from. These tracks are provided to our users with the intent that they assist in interpretation of other data, but must be used with caution. Not all annotations remain intact when lifted in this manner and in any case, cannot by definition contain any sequence that is new to the newer assembly. It should also be noted that tracks containing large regions will not lift as well because of the increased chance of spanning a region that has changed between the two assemblies. 
/goldenPath/help/hic.html:Genome_Browser_hic_Track_Format	hic Track Format Note: The UCSC tools currently support hic versions 6-8. The hic track configuration help page describes hic display options. Hic format is an indexed binary format designed to permit fast random access to contact matrix heatmaps. The format was designed by the Aiden Lab at Baylor College of Medicine. More information on the hic format itself can be found in the documentation on Github. The format is used for displaying chromatin conformation data in the browser. This format is useful for displaying interactions at a scale and depth that exceeds what can be easily visualized with the interact and bigInteract formats. After running a chromatin conformation experiment such as in situ Hi-C, researchers can pass their results through the Juicer pipeline to produce a hic file. Due to the large size of many of these files, UCSC is not able to support direct upload via our Custom Track interface. Instead, users should place their Hic files in a web-accessible space and enter the URL as a bigDataUrl. If you do not have access to a web-accessible server and need hosting space for your Hic files, please see the Hosting section of the Track Hub Help documentation. [hic draw modes] Generating a hic track The typical workflow for generating a hic custom track is this: 1. Prepare your data by processing it with the Juicer pipeline to create a file in the hic format. 2. Move the hic file (my.hic) to an http, https, or ftp location. 3. Construct a custom track using a single track line. The basic version of the track line will look something like this: track type=hic name="My HIC" bigDataUrl=http://myorg.edu/mylab/my.hic 4. Paste the custom track line into the text box in the custom track management page, click "submit" and view in the Genome Browser. Parameters for hic custom track definition lines All options are placed in a single line separated by spaces (lines are broken only for readability here): track type=hic bigDataUrl=http://... name=track_label description=center_label visibility=display_mode db=db Note if you copy/paste the above example, you must remove the line breaks. Click here for a text version that you can paste without editing. The track type and bigDataUrl are REQUIRED: type=hic bigDataUrl=http://myorg.edu/mylab/my.hic The remaining settings are OPTIONAL: name track label # default is "User Track" description center label # default is "User Supplied Track" visibility full|dense|hide # default is hide (will also take numeric values 4|1|0) db genome database # e.g. hg19 for Human Feb. 2009 (GRCh37) Note that hic tracks currently only support the full, dense, and hide visibility modes. The hic track configuration help page describes the hic track configuration page options. Example #1 In this example, you will create a custom track for a hic file that is already on a public internet server — data from an in situ Hi-C experiment on the HMEC cell line mapped to the hg19 assembly (Rao et al., 2014). The line breaks inserted in the track line for readability must be removed before submitting this entry as a Custom Track. Click here for a text version you can paste without editing. The "browser" line above is used set the default view to a region of chromosome 21. browser position chr21:32,000,000-35,000,000 track type=hic name="hic Example One" description="hic Ex. 1: in situ Hi-C on HMEC" db=hg19 visibility=dense bigDataUrl=http://hgdownload.soe.ucsc.edu/gbdb/hg19/bbi/hic/GSE63525_HMEC_combined.hic Paste the "browser" line and "track" line into the custom track management page for the human assembly hg19 (Feb. 2009), then click the "submit" button. On the following page, click the "chr21" link in the custom track listing to view the hic track in the Genome Browser. Example #2 In this example, you will load a hub that has hic data described in a hub's trackDb.txt file. First, navigate to the Basic Hub Quick Start Guide and review an introduction to hubs. Visualizing hic files in hubs involves creating three text files: hub.txt, genomes.txt, and trackDb.txt. The browser is passed a URL to the top-level hub.txt file that points to the related genomes.txt and trackDb.txt files. The trackDb.txt file contains stanzas for each track that outlines the details and type of each track to display, such as these lines for a hic file located at the bigDataUrl location: track hic1 bigDataUrl http://http://hgdownload.soe.ucsc.edu/gbdb/hg19/bbi/hic/GSE63525_GM12878_insitu_primary+replicate_combined.hic shortLabel hic example longLabel This hic file shows in situ Hi-C data from Rao et al. (2014) on the GM12878 cell line type hic visibility dense Note: there is now a useOneFile on hub setting that allows the hub properties to be specified in a single file. More information about this setting can be found on the Genome Browser User Guide. Here is a direct link to the trackDb.txt file to see more information about this example hub, and below is a direct link to visualize the hub in the browser, where this example hic file displays in dense mode alongside the other tracks in this hub. You can find more Track Hub hic display options on the Track Database (trackDb) Definition Document page. http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg19&hubUrl=http://genome.ucsc.edu/goldenPath/help/examples/hubDirectory/hub.txt [Ex hic image from track hub] Sharing Your Data with Others If you would like to share your hic data track with a colleague, there are a couple of options. One method is to load your data track into the UCSC Genome Browser and then create a saved session by following the instructions here. If you are looking for a more automated method of sharing data, you may be interested to learn how to create a direct URL that loads custom data files. For a demonstration of this, see Example 6 on the custom tracks page. 
/goldenPath/help/hgGenomeHelp.html:Genome_Graphs	Genome Graphs User's Guide Contents Introduction Formatting, uploading and importing data - Formatting data - Uploading data - Importing data Quick start Displaying data in Genome Graphs - Configuring the display - Setting a significance threshold - Setting a data region Viewing data in the Genome Browser Viewing data in the Gene Sorter Deleting data Correlating data ------------------------------------------------------------------------ Questions and feedback on this User's Guide are welcome. User questions and answers on Genome Graphs and other topics are available in the Genome Browser mailing list. Introduction Genome Graphs is a tool for displaying genome-wide data sets such as the results of genome-wide SNP association studies, linkage studies and homozygosity mapping. Using the Genome Graphs tool, you can: - upload several sets of genome-wide data and display them simultaneously - click on an area of interest and go directly to the genome browser at that position - set a significance threshold for your data and view only regions that meet that threshold - view the genes that exist in areas where your data meet your significance threshold To return to Genome Graphs from any other location on the Genome Browser website, use your browser's Back button, or click Home on the blue navigation bar, then click the Genome Graphs link. Note that only the "standard" chromosomes are displayed in the Genome Graphs display; haplotype and mitochondrial chromosomes are not displayed. This User's Guide is aimed at both the novice Genome Graphs user as well as the advanced user. If you are new to the Genome Graphs tool, read the Quick Start section to learn about the basics using some sample data. Advanced users may want to proceed directly to the section that addresses a particular area of functionality in detail. Formatting, uploading and importing data Formatting data Genome Graphs allows you to upload data from files that reside on your computer. Several file formats are accepted by the program. For all formats there is a single line for each marker. Each line starts with information on the marker, and ends with the numerical values associated with that marker. The markers can be of one of the following types: - chromosome base: e.g., chr1 130000 (Note that the first base in a chromosome is considered position 0) - STS Marker: e.g., RH75228 - dbSNP rsID: e.g., rs12345 - Affymetrix 500k Gene Chip: e.g., SNP_A-1780270 - Affymetrix Genome-Wide SNP Array 6: e.g., SNP_A-8575125 - Affymetrix SNP Array 6 Structural-Variation: e.g., CN_47396 - Illumina HumanHap300 Bead Chip: e.g., rs3934834 - Illumina HumanHap550 Bead Chip: e.g., rs3094315 - Illumina HumanHap650 Bead Chip: e.g., rs3094315 - Agilent CGH 244A: e.g. A_14_P112718 The marker-value pairs in each line of the file can be separated with a single space, a tab, or a comma. The file can contain multiple values for each marker. In that case, a separate graph will be created for each value column in the input file. For example, chromosome base markers with only one value associated with the marker would be entered like this: chrX 100000 1.23 dbSNP rsID markers with two values associated with the marker would be entered like this: rs10218492 0.384 0.882 The Genome Graph program will map the marker IDs to the genome. In cases where the marker maps to more than one location in the genome, the value(s) in your input file will be associated with each location. If the value associated with your marker is positive, do not include a sign (e.g., "+"). Include a sign ("-") only if the value is negative. Note that markers can only be mapped to assemblies for which there already exists a track of the type that contains your marker type. You can not, for example, use dbSNP rsID markers for the cow genome, as it does not have a SNP track. Uploading data Once you have created your input file, you must upload it to Genome Graphs. From the main Genome Graphs page, choose the clade, genome, and assembly to which your data pertains. If you are unsure of the UCSC assembly name, check this page. Then, click the upload button to go to the upload page. To upload a file in any of the supported formats, locate the file on your computer using the controls next to "file name", and then submit. The other controls on this form are optional, but can be used to enhance the display. In general, the controls that default to "best guess" will not need modification, since the default guess is almost always correct. The controls for display min and max values and connecting lines may be adjusted later via the configuration page. Here is a description of each control: - name of data set: Displayed in graph drop-down in Genome Graphs and as the track name in Genome Browser. Only the first 16 characters are visible in some contexts. For data sets with multiple graphs, this is the first part of the name, shared with all members of the data set. - description: A short sentence describing the data set. Displayed in the Genome Graphs and Genome Browser configuration pages, and as the center label in the Genome Browser. - file format: Controls whether the upload file is a tab-separated, comma-separated, or space separated table. - markers are: Describes how to map the data to chromosomes. The choices are that either the first column of the file is an ID of some sort, or the first column is a chromosome and the next a base. The IDs can be SNP rs numbers, STS marker names or IDs from any of the supported genotyping platforms. - column labels: Controls whether the first row of the upload file is interpreted as labels or data. If the first row contains text in the numerical fields, or if the mapping fields are empty, it is interpreted by "best guess" as labels. This is generally correct, but you can override this interpretation by explicitly setting the control. - display min value/max value: Set the range of the data set that will be plotted. If left blank, the range will be taken from the min/max values in the data set itself. For all data sets to share the same scale, you will usually need to set this. - label values: A comma-separated list of numbers for the vertical axis. If left blank, the axis will be labeled at the 1/3 and 2/3 points of your data range. - draw connecting lines: Lines are drawn connecting data points that are separated by this number of bases or fewer. - file name, or Paste URLs or data: Specify the uploaded data -- enter either a file on your local computer; or a URL at which the data file can be found; or simply paste-in the data. If entries are made in both fields, the file name will take precedence. Importing data In addition to supplying your own genome-wide data files, you can also import existing database tables from an assembly into the Genome Graphs tool. Any table containing positional information can be imported. This includes tables of the following types: BED, PSL, wiggle, MAF, and bedGraph. Custom track tables can be imported as well. The tables made by Genome Graphs (chromGraph) can not be imported as they are already in the format used by the tool, thus no conversion is necessary. All tables imported into Genome Graphs will be converted into a custom track of type chromGraph using a window-size of 10,000 bases. To import a table or custom track, choose the group, track, and table from the lists, then click the submit button. The other controls are optional, though completing them will enhance the display. The controls for display min and max values and connecting lines can be set later via the configuration page as well. Here is a description of each control. - name of data set: This will be displayed in the graph list in the Genome Graphs tool and as the track name in the Genome Browser. Only the first 16 characters are visible in some contexts. For data sets with multiple graphs, this is the first part of the name, shared with all members of the data set. - description: Enter a short sentence describing the data set. It will be displayed in the Genome Graphs tool and in the Genome Browser. - display min value/max value: Set the range of the data set to be plotted. If left blank, the range will be taken from the min and max values in the data set itself. If you would like all of your data sets to share the same scale, you will need to set this. - label values: A comma-separated list of numbers for the vertical axis. If left blank the axis will be labeled at the 1/3 and 2/3 point. - draw connecting lines: Lines connecting data points separated by no more than this number of bases are drawn. - depth or coverage: When importing positional tables, you can choose to convert those tables to the chromGraph format by using either the depth or coverage conversion method. Both conversion methods use a non-overlapping window size of 10,000 bases when converting to the chromGraph format. In the depth method, the weighted average for each 10,000 base window is assigned to a single point in the center of this window. Whereas the coverage method is binary &mdash if there is even one point in the input table in that 10,000 base window, the resulting graph will have a value of 1 for that range. Quick start Use the examples in this section of the User's Guide to get a feel for how the tool works. Refer to other sections in this User's Guide for details and instructions for more advanced features. The Genome Graphs tool comes pre-loaded with sample data. These sample data sets are from real-world genome-wide studies. Use these data sets to quickly see what the tool looks like when data is displayed. To view the sample data, choose a data set from the graph drop-down list, then choose your desired display color from the in drop-down list. The tool will display the data set directly above the chromosomes in Genome Graphs. Read on to learn how to customize the display. Example #1 — SNPs on chr22 Follow these steps to display in Genome Graphs all of the highest quality SNPs on chromosome 22 for the hg18 assembly whose predicted functional role is "coding non-synonymous" (where there is a change in the peptide for the allele with respect to the reference assembly). Note that there are no SNPs on the p-arm of chromosome 22. This data set is formatted in the "marker value" style. The markers are dbSNP rsIDs. The associated value is +1 if the SNP is on the positive strand, and -1 if the SNP is on the negative strand. Here are the first ten rows of the data file: rs1007298 1 rs1007863 1 rs10154509 1 rs10154678 1 rs10154785 1 rs1018448 1 rs10212022 1 rs1022478 1 rs1042311 1 rs1042435 1 Step 1. Upload the data into the Genome Graphs tool Copy the entire sample data set into a text editor and save the file to your computer. This data set is associated with the human assembly: hg18 (Mar. 2006). Be sure to configure the Genome Graphs tool to use the hg18 assembly like so: clade: Vertebrate genome: Human assembly: Mar. 2006 Upload the file into the Genome Graphs tool. You can configure each control on the upload page, or just leave them set to their default values. The upload process may take some time, as the program is actually mapping each rsID in the input file to its location(s) in the genome. Step 2. Display the graph in Genome Graphs Now that your input file has been uploaded to the server, you will want to display it in the Genome Graphs tool. To display your uploaded data, simply choose the graph name from the graph drop-down list, then choose your desired display color from the in drop-down list. Your graph will be displayed directly above the chromosomes in Genome Graphs. You should see the data plotted directly above chromosome 22. Step 3. View the graph in the Genome Browser From the Genome Graphs display, click anywhere on the graph or on chromosome 22 to open the Genome Browser for hg18 centered at that location on chr22. The graph will be drawn as a track near the top of the Genome Browser display. Displaying data in Genome Graphs Once you have uploaded your data, you will want to display it in the Genome Graphs tool. To display your uploaded data, simply choose the graph name from the graph drop-down list, then choose the color in which you would like it to be displayed from the in drop-down list. Your graph will be displayed directly above the chromosomes in Genome Graphs. Read on to learn how to customize the display. Configuring the display Configuring the graphs display To go to the configuration page, click the configure button on the main Genome Graphs page. This is the page from which you can configure many overall aspects of the Genome Graphs display. Individual graphs can also be configured (see the next section for help on that). On this page you will find the following controls: - image width - controls the overall width of the graphs display on the main Genome Graphs page. The default is 620 pixels. - graph height - controls the height of the graph(s) in the space above each chromosome. The default is 27 pixels. - graphs per line - controls how many graphs are displayed on each line in the space above each chromosome. For example, if you set this value to two, the display will superimpose two graphs on top of each other on one line. The axis label for the first graph will appear on the left side of the display and the axis for the second graph on the right side. - lines of graphs - controls how many sets of graphs will appear above each chromosome. For example, if you set this value to 2, the display will make room for two lines of graphs (each at the graph height above) in the space above each chromosome. - chromosome layout - controls how the chromosomes are laid out in the Genome Graphs display. You can choose to view one or two chromosomes on each horizontal line in the display. Alternatively, you can set up the display such that all of the chromosomes appear in one long line. If you choose this layout, you may want to adjust the width of the image (image width above). - numerical labels - check this box if you would like to see axis labels to the right/left of the display. If you did not specify label values when you uploaded your file, the numerical labels will default to 1/3 and 2/3 of the max and min values in your data input file. - highlight missing - check this box if you would like to see the areas in your graph where there is no data. Note that if you are displaying more than one graph, this attribute only pertains to the first graph. - region padding - controls the size of the data regions. The data points in your graphs which exceed the significance threshold are padded by this number of bases on either side. The default places 25,000 bases on each side. When you have completed configuring the display, click the submit button to return to the Genome Graphs display. Configuring individual graphs Near the bottom of the Configuration page, you will see a list of the graphs that you have uploaded. Click on the hyperlinked graph name to configure that graph. This configuration pertains to the Genome Graphs view. You can set the range of the display by editing the display min/max value values. This will restrict the Genome Graphs display for this graph to that data range. The axis will be labeled at 1/3 and 2/3 of the data range that you set. If your data is sparse, you may want to draw lines between your data points. You can configure that by editing the draw connecting lines between markers separated by up to ... bases value. The default value is 25,000,000 bases. When you have completed configuring the display, click the submit button twice to return to the Genome Graphs display. Setting a significance threshold Most genome-wide data has some amount of noise and is only interesting when the data values are above a certain value. You can set this value using the significance threshold input box. Enter a decimal number in this input box and click Enter. The display will now have a light gray line across the graph at this data value. If you have more than one graph displayed, the significance threshold only pertains to the graphs that contain the significance threshold in the displayed data range. The significance threshold works in concert with the browse regions and sort genes buttons; it will affect the regions that are displayed once you click either of these two buttons. To open the Genome Browser with a view of all of the regions in your graph that include data points that pass the significance threshold, click the browse regions button. This will open the Genome Browser with a navigation pane on the left side of the screen. This pane will contain links to all regions which pass your significance threshold. Note that if you are displaying more than one graph, the significant regions are based only on the first graph in the display list. To view a list of genes which are in regions that pass the significance threshold, click the sort genes button. This will open the Gene Sorter with only the genes that are in significant locations with respect to your data. If you would rather view all of your regions without restricting the output to only those regions that pass the significance threshold, simply delete any values from the significance threshold input box and click Enter before clicking browse regions. Setting a data region The data region is the span of bases that will be added to either side of the data points in your graphs which exceed the significance threshold. Set the data region by editing the region padding value on the configuration page. The combination of setting the data region and the significance threshold will affect two things: - the regions displayed in the Genome Browser after you click the browse regions button, - the genes displayed in the Gene Sorter after you click the sort genes button. For example, take a data set that contains the following data: chr2 100100000 2.3 chr2 100100500 4.5 chr2 100101000 1.2 If you set the significance threshold at 4.0, one data point in the data set passes that threshold. If you then set the data range to 200, then the one significant data point will be padded on each side by 200 base pairs. In that case, the only resulting significant data region will be chr2:100,100,300-100,100,700. If instead you set the data range to 2,000, then the one significant data point will be padded on each side by 2,000 base pairs. In that case, the resulting significant data region will be chr2:100,098,500-100,102,500. Viewing data in the Genome Browser To view your graphs in the Genome Browser, click the browse regions button. This will open the Genome Browser with your graph(s) displayed as track(s). You can configure and edit your track as you can any other track in the Genome Browser. In addition to the Genome Browser, you will also see a pane on the left-hand side, which contains links to all of the significant regions in your data. Please note that if you are displaying more than one graph in Genome Graphs, the significant regions are based only on the first graph in the display list. You can also navigate to the Genome Browser by clicking directly on a graph or chromosome in Genome Graphs. The Genome Browser will open with a 1,000,000 bp window centered on the location on which you clicked. Viewing data in the Gene Sorter To view the set of genes that are in significant regions in your data, click the sort genes button. This will open the Gene Sorter with a filter to include only genes that are located in regions in your input data that are above the significance threshold. Please note that if you are displaying more than one graph in Genome Graphs, the significant genes are based only on the first graph in the display list. If the graph was uploaded using markers, then a custom Gene Sorter column with the same name as the graph will be created. This column will list all markers for each gene that contain values above the significance threshold. Deleting data There are several ways to delete your data once it has been uploaded. If you are viewing your data as a track in the Genome Browser, you can click on the mini-button or track control for the track and delete the track using the Remove custom track button. You can also choose to reset your cart which will reset the browser interface settings to their defaults, as well as delete all custom tracks and data. Do this by clicking the "Reset All User Settings" under the top blue Genome Browser menu. Your data will be saved on our server for at least 48 hours from the time you last access it, unless it is saved in a Session. Correlating data sets To calculate how well correlated with one another your data sets are, click the correlate button. This will calculate and display the correlation coefficient (R) among each of your data sets. R, also known as Pearson's correlation coefficient, is a measure of the extent that two graphs move together. The value of R ranges between -1 and 1. A positive R indicates that the graphs tend to move in the same direction, while a negative R indicates that they tend to move in opposite directions. R-Squared (which is indeed just R*R) measures how much of the variation in one graph can be explained by a linear dependence on the other graph. R-Squared ranges between 0 when the two graphs are independent to 1 when the graphs are completely dependent. To return to the Genome Graphs, click the return to graphs button. 
/goldenPath/help/hgCollectionHelp.html:Track_Collection_Builder_Help	Track Collection Builder Help 
/goldenPath/help/iupac.html:Genome_Browser_IUPAC_Codes	IUPAC codes The International Union of Pure and Applied Chemistry (IUPAC) has defined a standard representation of DNA bases by single characters that specify either a single base (e.g. G for guanine, A for adenine) or a set of bases (e.g. R for either G or A). UCSC uses these single character codes to represent multiple observed alleles of single-base polymorphisms. Symbol Bases Origin of designation -------- ------------------ ------------------------------------ G G Guanine A A Adenine T T Thymine C C Cytosine R G or A puRine Y T or C pYrimidine M A or C aMino K G or T Keto S G or C Strong interaction (3 H bonds) W A or T Weak interaction (2 H bonds) H A or C or T not-G, H follows G in the alphabet B G or T or C not-A, B follows A V G or C or A not-T (not-U), V follows U D G or A or T not-C, D follows C N G or A or T or C aNy 
/goldenPath/help/hubQuickStartFilter.html:Track_Hub_Quick_Start_Filter	Track Hub Filters Quick Start Guide Track Hubs are a method of displaying remotely-hosted annotation data quickly and flexibly on any UCSC assembly or remotely-hosted sequence. There are different filtering options available for bigBed files depending on the kind of data the filter will be applied to. These filters are also described in the trackDb help doc. Note: for configurable features, like filters, an additional period "." or plus "+" is required in the type declaration, for instance type bigBed 5 . or type bigBed 9 +. Contents filter.fieldName - Used for numerical data filterText.fieldName - Used for text filtering filterValues.fieldName - Used for filtering by prespecified values or categories in data filter.fieldName filter.fieldName is used to enable numerical filtering within a field or column. It is often seen as a filter on data that contains a score field. It requires a default parameter to be passed, often this parameter is 0. By default, the range of values will be 0 to 1000. However, this range can be modified with the filterLimits.fieldName parameter. Additionally, the filter can be modified to take in a range of values with the filterByRange.fieldName on parameter. For more information on filter.fieldName, see the trackDb help doc entry. filter Example 1 In this first example, we have a simple track with 10 items. The data looks as follows: chr7 127000000 127000005 1 1 chr7 127000010 127000015 2 2 chr7 127000020 127000025 3 3 chr7 127000030 127000035 4 4 chr7 127000040 127000045 5 5 chr7 127000050 127000055 6 6 chr7 127000060 127000065 7 7 chr7 127000070 127000075 8 8 chr7 127000080 127000085 9 9 chr7 127000090 127000095 10 10 In this case, we have duplicated the name and score fields for clarity. We will be applying a default filter of 4 to the score field. This will mean that by default only items 4-10 will display. This filter is enabled with the line filter.score 4. The hub.txt looks as follows: track filterScore4 shortLabel filter.fieldNameDefault 4 longLabel Numerical filter with a default value of 4 passed visibility pack type bigBed 5 . filter.score 4 bigDataUrl example1.bb Below are all the materials for example 1: - bed file - bigBed file - .as file - hub.txt file - Example session The example session will display an image like the following, which hides items 1-3 and displays items 4-10. The three filtered items are also noted on the longLabel above the track as (3 items filtered). If we just want to enable filtering, but pass no default value, we can use filter.score 0. [Numerical filter enabled on bigBed] filter Example 2 In this second example we have four tracks with filter.fieldName to allow numerical filtering, filterByRange.fieldName on to enable range filtering, and filterLimits.fieldName to designate upper and lower range boundaries. filter.fieldName will be used on two separate fields (score and name) to demonstrate multiple filters. Lastly, filterLabel.fieldName will be used to change the default filter message to instead "Value range to filter" for the score field. The data used is the same as example 1 above: chr7 127000000 127000005 1 1 chr7 127000010 127000015 2 2 chr7 127000020 127000025 3 3 chr7 127000030 127000035 4 4 chr7 127000040 127000045 5 5 chr7 127000050 127000055 6 6 chr7 127000060 127000065 7 7 chr7 127000070 127000075 8 8 chr7 127000080 127000085 9 9 chr7 127000090 127000095 10 10 The data are organized as a bed5. Standard filtering will be enabled in the 4th name field, and filtering by ranges on the 5th score field. The score field will also be given a custom label with the filterLabel parameter. Here is an example of the first track stanza: track filteringByRangeAllValues shortLabel filteringByRangeDefault longLabel Filter by range enabled with default score including all values visibility pack type bigBed 5 . filter.name 0 filter.score 0:10 filterByRange.score on filterLimits.score 0:10 filterLabel.score Value range to filter bigDataUrl example2.bb Below are all the materials for example 2: - bed file - bigBed file - .as file - hub.txt file - Example session Enabling range filters for the score field as well as the standard filter for the name field, we see the following options in the track description page. Any number of filters can be enabled on a track simultaneously. [] Going to the example session will display a browser image like so: [Range filters enabled on bigBed] Each of the tracks is filtering by a different value, and in the final track two separate filters are enabled. - The first track (5-10) displays only items with scores 5-10. - The second track (3-8) displays only items with scores 3-8. - The third track has two filters. A range filter (3-8) and a second filter (5). Only the values that pass both filters are seen. - The fourth track has a filter inclusive of all values passed(0-10), so all items are displayed. filter Example 3 This third example explores how the numerical filters interact with non-numerical characters. The data is comprised of 10 items as a bed5, with the same coordinates as the examples above, a name field, and an arbitrary score field. The name field contains a mix of numerical and non-numerical characters. chr7 127000000 127000005 0 0 chr7 127000010 127000015 -1 0 chr7 127000020 127000025 2% 0 chr7 127000030 127000035 -3 0 chr7 127000040 127000045 4 0 chr7 127000050 127000055 5n 0 chr7 127000060 127000065 5 0 chr7 127000070 127000075 NA 0 chr7 127000080 127000085 . 0 chr7 127000090 127000095 <> 0 Filtering will be enabled on the 4th name field. The trackDb stanza looks as follows: track filteringNonNumerical shortLabel filteringNonNumerical longLabel Using numerical filters on a field with both numerical and non-numerical values visibility pack type bigBed 5 . filter.name 0 bigDataUrl example3.bb Below are all the materials for example 2: - bed file - bigBed file - .as file - hub.txt file - Example session The filter is being passed on the name field, with a default value of 0. The example session will show which items still display with this default value: [Filtering on non-numerical characters] The only items being filtered are the negative values, -1 and -3. Entirely non-numerical characters are interpreted as 0. If we instead change the filter to be 2, we see the following: [Filtering on non-numerical characters] In this case we see the items that start with non-numerical characters get filtered. Items that start with a number, and are following by another character, are treated as the number. This can be seen with the 2% value, which remains visible with the filter active. It is important to keep in mind that non-numerical values, such as NA, will be visible when the default 0 filter is active, but will be removed when any positive numerical filter is activated. filterText.fieldName filterText.fieldName is used to enable text searching in the specified fieldName. This will display any items passed which match exactly the searched term, or only part of the search term. Two types of searching are supported, wildcard searching (*) or regular expression searching (regexp). The mode between the two types can be freely changed in the track description page, or a default passed using the filterType.fieldName parameter. Lastly, the filter label will be the description of the field as specified by the autoSql (.as) file. This label can be customized with the filterLabel.fieldName parameter. A value can be passed with this setting to enable a specific filter by default. Also, the default search type is wildcard (*). For more information on filterText.fieldName, see the trackDb help doc entry. filterText Example 1 In this first example, we have 10 genes with arbitrary coordinates as follows: chr7 127000000 127000005 EGFR 1 chr7 127000010 127000015 VEGFA 2 chr7 127000020 127000025 APOE 3 chr7 127000030 127000035 IL6 4 chr7 127000040 127000045 TGFBI 5 chr7 127000050 127000055 BRCA1 6 chr7 127000060 127000065 BRCA2 7 chr7 127000070 127000075 MTHFR 8 chr7 127000080 127000085 ESR1 9 chr7 127000090 127000095 AKT1 10 By default, we would like our data to display only BRCA1 and BRCA2. The easiest way to accomplish this is to enable a filterText wildcard filter. We will be filtering on the name field using the following setting: filterText.name BRCA* The hub.txt looks as follows: track filterTextDefaultBRCA shortLabel filterTextBRCA longLabel Wildcard filterText with default BRCA value visibility pack type bigBed 5 . filterText.name BRCA* bigDataUrl filterTextExample1.bb Below are all the materials for example 1: - bed file - bigBed file - .as file - hub.txt file - Example session The example session will display an image like the following, showing only the BRCA items. The eight filtered items are also noted on the longLabel above the track as (8 items filtered). If we just want to enable filtering, but pass no default value, we can use filterText.name *. [filterText filter enabled on bigBed] We can also change the filter type or filter value (or remove values) by going to the track description page, which will show the following: [filterText filter track description page] Changing the filter from BRCA* to *A* would expand the wildcard match to all items with A. See the trackDb help doc entry for additional information including an example using regexp. For instance, with wildcard changed to a regexp type of search, putting in .*A\|B.* will match any items with an A or B in it, while .*[0-9] will match any item ending in a number. filterValues.fieldName filterValues.fieldName is used to enable filtering by pre-specified values within a field. It can be used on fields that can contain one text value or a list of comma-separated values of text, like "classA,classB". Usually, these are category names. The option requires at least one value to filter on. Every individual possible value that can ever occur in the field must be passed in a comma separated list. You will then be able to select those values as categories, choosing to display only items that belong to one, any, or at least one of the selected values. By default, the user can select multiple values from this list and the filter lets pass any features with at least one of these values (multipleListOr). The type of selection can be designated by passing the optional filterType.fieldName parameter. Possible options are: - single - Allows selection of a single item from dropdown menu. - singleList - Allows selection of a single item from dropdown menu. Accepts comma-separated values in the bigBed field. - multiple - Allows selection of any number of items from dropdown menu. Shows items that contain at least one of the selections. - multipleListOr - Allows selection of any number of items from dropdown menu. Shows items that contain at least one of the selections. Accepts comma-separated values in the bigBed field. Enables the radio button that allows swapping of filter type in the track description page. - multipleListOnlyOr - Same as multipleListOr except it disables the option to change the filter type from the browser interface. - multipleListAnd - Allows selection of any number of items from dropdown menu. Displays only items containing all of the selections. Accepts comma-separated values in the bigBed field. Enables the radio button that allows swapping of filter type in the track description page. - multipleListOnlyAnd - Same as multipleListAnd except it disables the option to change the filter type from the browser interface. As with other filters, default values can be passed to filterValues using the filterValuesDefault.fieldName parameter. It can take a comma-separated list just like filterValues.fieldName, and any items included will be automatically selected. The labels in the menu shown to the user can be configured to display a different name/label than the one present in the bigBed field. This can be helpful when the data values are written in short form, but you want a longer more descriptive name to show up in the UI. The format for this substitution is as follows: filterValues.fieldName fieldValue1|alternativeName1,fieldValue2|alternativeName2... In this example, a bigBed could have value AML in a field called Disease, but we would like the menu to display Acute Myeloid Leukemia so the line could be filterValues.Disease AML|Acute Myeloid Leukemia,MSC|Melanoma Skin Cancer... where the filter display would have the full disease names, while the data instead in the bigBed was the abbreviation. This can also be used to reduce the size of bigBed files. The following session contains a hub with example tracks of all the possible filterValues settings. It can be used to explore the differences and restrictions of each of the settings. filterValues Example 1 In this first example, we have a small track with 10 items. The score and names of the items are the same, and there is an additional field added which labels the score as either an even or odd number, making our file a bigBed 5+1: chr7 127000000 127000005 1 1 odd chr7 127000010 127000015 2 2 even chr7 127000020 127000025 3 3 odd chr7 127000030 127000035 4 4 even chr7 127000040 127000045 5 5 odd chr7 127000050 127000055 6 6 even chr7 127000060 127000065 7 7 odd chr7 127000070 127000075 8 8 even chr7 127000080 127000085 9 9 odd chr7 127000090 127000095 10 10 even We wish to add a filter which allows selection on whether the item is even or odd. This is enabled by the line filterValues.OddEven odd,even. Remember that all possible values need to be listed. We also wish for only a single selection to be possible, which we will do with the following setting filterType.OddEven singleList. Below is the track stanza in the hub.txt: track filteValuesOddEven shortLabel filterValues.OddEven odd,even longLabel filterValues categorical filter on odd and even values visibility pack type bigBed 5 + 1 filterValues.OddEven odd,even filterType.OddEven singleList bigDataUrl filterValuesExample1.bb Below are all the materials for this example: - bed file - bigBed file - .as file - hub.txt file - Example session filterValues Example 2 In this second example, we have signal data on 10 items, and want to enable filtering on any number of annotation type. Note that the annotation type is written in shorthand: chr7 127000000 127000005 signal1 0 DNA-BR,AH chr7 127000010 127000015 signal2 0 AS,BS chr7 127000020 127000025 signal3 0 BS chr7 127000030 127000035 signal4 0 BS chr7 127000040 127000045 signal5 0 DNA-BR chr7 127000050 127000055 signal6 0 DNA-BR,BS chr7 127000060 127000065 signal7 0 AS chr7 127000070 127000075 signal8 0 BS chr7 127000080 127000085 signal9 0 AS,AH chr7 127000090 127000095 signal10 0 DNA-BR,BS In this case, the annotations (the sixth annotationType column) represent the following: - DNA-BR - DNA-binding region - AS - active site - AH - alpha helix - BS - beta strand Some signals contain more than one type of annotation. We will enable the categorical filter using the filterValues.annotationType setting, including all the possible values. We will also substitute the complete annotation type names by using the pipe "|" character. We work with researchers most interested in DNA-binding regions that are also beta strands, so we will want the default behavior to display only those items. For this we will use filterType.annotationType multipleListAnd to require matching on all selections, and filterValuesDefault.annotationType DNA-BR,BS to pass the default values. Note that we do not use the type multipleListOnlyAnd, as that would not allow users to change selection type. Below is the complete track stanza of the hub.txt file: track filteValuesAnnotationType shortLabel filterValues.annotationType longLabel filterValues categorical filter on multiple annotation types visibility pack type bigBed 5 + 1 filterValues.annotationType DNA-BR|DNA-binding region,AS|active site,AH|alpha helix,BS|beta strand filterType.annotationType multipleListAnd filterValuesDefault.annotationType DNA-BR,BS bigDataUrl filterValuesExample2.bb Below are all the materials for this example: - bed file - bigBed file - .as file - hub.txt file - Example session Going to the example session will result in a browser image as such: [filterValues filter enabled on bigBed] Note that only signal6 and signal10 are displayed by default. That is because they are the only two items that have both DNA-BR (DNA-binding region) and BS (beta strand) as their type of annotation. If we click into the track, we see additional filter options. In the image below we have switched the filter to one or more match, meaning that if the items contain any of the selected items they will display. We have also expanded the selection to include AH (alpha-helix). Lastly, note that the menu displays the alternative long names instead of the shorthand: [filterValues filter enabled on bigBed] This will result in all items being displayed except for signal 7. This is because it is the only item that contains none of the selected categories, with its only annotation being AS (active site): [filterValues filter enabled on bigBed] Additional Resources - Track Hub User Guide - Guide To useOneFile setting - Search file .ix documentation - Mailing list question with searchable Track Hub - Mailing list question with searchable Custom Tracks - Track Database (trackDb) searchTrix Definition - Quick Start Guide to Organizing Track Hubs into Groupings - Quick Start Guide to Assembly Track Hubs 
/goldenPath/help/bigBed.html:Genome_Browser_bigBed_Track_Format	bigBed Track Format The bigBed format stores annotation items that can be either a simple or a linked collection of exons, much as BED files do. BigBed files are created from BED type files using the program bedToBigBed. The resulting bigBed files are in an indexed binary format. The main advantage of the bigBed files is that only those portions of the files needed to display a particular region are transferred to the Genome Browser server. Because of this, bigBed has considerably faster display performance than regular BED when working with large data sets. The bigBed file remains on your local web-accessible server (http, https, or ftp), not on the UCSC server, and only the portion that is needed for the currently displayed chromosomal position is locally cached as a "sparse file". If you do not have access to a web-accessible server and need hosting space for your bigBed files, please see the Hosting section of the Track Hub Help documentation. Additional indices can be created for the items in a bigBed file to support item search in track hubs. See Example #3 below for an example of how to build an additional index. See this wiki page for help in selecting the graphing track data format that is most appropriate for your type of data. To see an example of turning a text-based bedDetail custom track into the bigBed format, see this How to make a bigBed file blog post. Note that the bedToBigBed utility uses a substantial amount of memory: approximately 25% more RAM than the uncompressed BED input file. Quickstart example commands It is not hard to create a bigBed file. The following UNIX commands create one on a Linux machine (swap macOSX for linux for an Apple environment). The steps are explained in more detail in the following sections on this page: wget https://genome.ucsc.edu/goldenPath/help/examples/bedExample.txt wget https://genome.ucsc.edu/goldenPath/help/hg19.chrom.sizes wget http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/bedToBigBed chmod a+x bedToBigBed ./bedToBigBed bedExample.txt hg19.chrom.sizes myBigBed.bb mv myBigBed.bb ~/public_html/ The last step assumes that your ~/public_html/ directory is accessible from the internet. This may not be the case on your server. You may have to copy the file to another server and web-accessible location at your University. Once you know the URL to the file myBigBed.bb, you can paste this URL into the custom track box on the UCSC Genome Browser to display the file. Creating a bigBed track To create a bigBed track, follow these steps (for concrete Unix commands, see the examples below on this page): Step 1. Create a BED format file following the directions here. When converting a BED file to a bigBed file, you are limited to one track of data in your input file; therefore, you must create a separate BED file for each data track. If your BED file was originally a custom track, remove any existing "track" or "browser" lines from your BED file so that it contains only data. Your file does not need to be sorted by chromosome name, but all entries for a single chromosome must be together and sorted by chromosome start position. If you're not sure if this is true for your BED file, it may be easiest to sort the file using the "-sort" option for bedToBigBed. Finally, if your BED files are large, they can be compressed using gzip (e.g. myTrack.bed.gz) and still read by bedToBigBed. Step 2. Download the bedToBigBed program from the binary utilities directory. Example #2 below shows the exact Unix command. The bedToBigBed program can be run with several additional options. Some of these, such as the -as and -type options, are used in examples below. The -type option, describes the size of the bigBed file, -type=bedN[+[P]], where N is an integer between 3 and 12 and the optional +[P] parameter specifies the number of extra fields, not required, but preferred. Describing the size of the bigBed file is needed for access to extra fields like name, itemRgb, etc. Examples:-type=bed6 or -type=bed6+ or -type=bed6+3 . For a full list of the available options, type bedToBigBed (with no arguments) on the command line to display the usage message. Step 3. Use the fetchChromSizes script from the same directory to create the chrom.sizes file for the UCSC database you are working with (e.g., hg19). If the assembly genNom is hosted by UCSC, chrom.sizes can be a URL like: http://hgdownload.soe.ucsc.edu/goldenPath/genNom/bigZips/genNom.chrom.sizes. Step 4. Use the bedToBigBed utility to create a bigBed file from your sorted BED file, using the input.bed file and chrom.sizes files created in Steps 1 and 3: bedToBigBed input.bed chrom.sizes myBigBed.bb The chrom.sizes file can also be a 2bit or a chromAlias bigBed file using the following command-line arguments: -sizesIs2Bit -- If set, the chrom.sizes file is assumed to be a 2bit file. -sizesIsChromAliasBb -- If set, then chrom.sizes file is assumed to be a chromAlias bigBed file or a URL to a such a file Step 5. Move the newly created bigBed file (myBigBed.bb) to a web-accessible http, https, or ftp location. At this point you should have a URL to your data, such as "https://institution.edu/myBigBed.bb", and the file should be accessible outside of your institution/hosting providers network. For more information on where to host your data, please see the Hosting section of the Track Hub Help documentation. Step 6. If the file name ends with a .bigBed or .bb suffix, you can paste the URL of the file directly into the custom track management page, click "submit" and view the file as a track in the Genome Browser. By default, the file name will be used to name the track. To configure the track name and descriptions, you must create a "track line", as shown in Example 1 Configuration Step 1. Alternatively, if you want to set the track labels and other options yourself, construct a custom track using a single track line. Note that any of the track attributes listed here are applicable to tracks of type bigBed. The most basic version of the track line will look something like this: track type=bigBed name="My Big Bed" description="A Graph of Data from My Lab" bigDataUrl=http://myorg.edu/mylab/myBigBed.bb Paste this custom track line into the text box on the custom track management page. Examples Example #1: Load an existing bigBed file In this example, you will load an existing bigBed file, bigBedExample.bb, on the UCSC http server. This file contains data on chromosome 21 on the human hg19 assembly. To create a custom track using this bigBed file: 1. Paste the URL http://genome.ucsc.edu/goldenPath/help/examples/bigBedExample.bb into the custom track management page for the human assembly hg19 (Feb. 2009). 2. Click the "submit" button. 3. On the next page that displays, click the "go" link. To view the data in the bigBed track in the Genome Browser navigate to chr21:33,031,597-33,041,570. Configuration You can customize the track display by including track and browser lines that define certain parameters: 1. Construct a track line that references the bigBedExample.bb file: track type=bigBed name="bigBed Example One" description="A bigBed file" bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigBedExample.bb 2. Paste the track line into the custom track management page, click the "submit" button. On the next page that displays, click the "go" link. To view the data in the bigBed track in the Genome Browser navigate to chr21:33,031,597-33,041,570. 3. With the addition of the following browser line with the track line you can ensure that the custom track opens at the correct position when you paste in the information: browser position chr21:33,031,597-33,041,570 track type=bigBed name="bigBed Example One" description="A bigBed file" bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigBedExample.bb Paste the browser and track lines into the custom track page, click the "submit" button and the "go" link to see the data. Example #2: Create a bigBed file from a BED file In this example, you will convert a sample BED file to bigBed format. 1. Save the BED file bedExample.txt to a server, ideally one that is accessible from the internet. (Steps 1 and 2 in Creating a bigBed track, above). wget https://genome.ucsc.edu/goldenPath/help/examples/bedExample.txt 2. Save the file hg19.chrom.sizes to your computer. It contains the chrom.sizes data for the human (hg19) assembly (Step 3, above). wget https://genome.ucsc.edu/goldenPath/help/hg19.chrom.sizes 3. If you use your own file, it has to be sorted, first on the chrom field, and secondarily on the chromStart field. You can use the utility bedSort available here or the following UNIX sort command to do this: sort -k1,1 -k2,2n unsorted.bed > input.bed 4. Download the bedToBigBed utility (Step 2, above). Replace "linux" below with "macOSX" if your server is a Mac. wget http://hgdownload.soe.ucsc.edu/admin/exe/linux.x86_64/bedToBigBed chmod a+x bedToBigBed 5. Run the utility to create the bigBed output file (Step 4, above): ./bedToBigBed bedExample.txt hg19.chrom.sizes myBigBed.bb 6. Place the bigBed file you just created (myBigBed.bb) on a web-accessible server (Step 5, above). mv myBigBed.bb ~/public_html/ At some Universities, this involves using the commands ftp, scp or rsync to copy the file to a different server, one that is accessible from the internet. We have documentation how to find such a server. 7. Paste the URL itself into the Custom Tracks entry form or construct a track line that points to your bigBed file (Step 5, above). 8. Create the custom track on the human assembly hg19 (Feb. 2009), and view it in the Genome Browser (Step 6, above). Note that the original BED file contains data on chromosome 21 only. Example #3: Create a bigBed file with extra (custom) fields BigBed files can store extra fields in addition to the predefined BED fields. In this example, you will create your own bigBed file from a fully featured existing BED file that contains the standard BED fields up to and including the color field called itemRgb (field 9), plus two additional non-standard fields (two alternate names for each item in the file). The standard BED column itemRgb contains an R,G,B color value (e.g. "255,0,0"). The resulting bigBed file will have nine standard BED columns and two additional non-standard user-defined columns. If you add extra fields to your bigBed file, you must include an AutoSql format (.as) file describing the fields. In this file, all fields (standard and non-standard) are described with a short internal name and also a human-readable description. For more information on AutoSql, see Kent and Brumbaugh, 2002, as well as examples of .as files in this directory. Then, the bedToBigBed program is run with the arguments -type=bed9+2 and also -as=bedExample2.as to help correctly interpret all the columns in the data. This example also demonstrates how to create an extra search index on the name field, and the first of the extra fields to be used for track item search. The searchIndex setting requires the input BED data to be case-sensitive sorted (sort -k1,1 -k2,2n), where newer versions of the tool bedToBigBed (available here) are enhanced to catch improper input. 1. Save the BED file bedExample2.bed to your computer (Steps 1 and 2 in Creating a bigBed track, above). 2. Save the file hg18.chrom.sizes to your computer. This file contains the chrom.sizes for the human (hg18) assembly (Step 4, above). 3. Save the AutoSql file bedExample2.as to your computer. This file contains descriptions of the BED fields, and is required when the BED file contains a color field. 4. Download the bedToBigBed utility (Step 3, above). 5. Run the utility to create a bigBed output file with an index on the name field and the first extra field: (Step 5, above): bedToBigBed -as=bedExample2.as -type=bed9+2 -extraIndex=name,geneSymbol bedExample2.bed hg18.chrom.sizes myBigBed2.bb 6. Paste the URL of the file into the custom tracks entry form, or alternatively construct a track line that points to your bigBed file (Step 7, above). Because this bigBed file includes a field for color, you must include the itemRgb attribute in the track line. It will look somewhat similar to this (note that you must insert the URL specific to your own bigBed file): track type=bigBed name="bigBed Example Three" description="A bigBed File with Color and two Extra Fields" itemRgb="On" bigDataUrl=http://yourWebAddress/myBigBed2.bb 7. Create the custom track on the human assembly hg18 (Mar. 2006), and view it in the Genome Browser (step 8, above). Note that the original BED file contains data on chromosome 7 only. 8. If you are using the bigBed file in a track hub, you can use the additional indices for track item searches. See the setting "searchIndex" in the Track Database Definition Document for more information. For example, if you run the bedToBigBed utility with the option -extraIndex=name, you will be able to search on the "name" field by adding the line searchIndex name to the stanza about your bigBed in the hub's trackDb.txt file. While searchIndex expects a search string with an exact match in the index, another setting for Track Hubs, searchTrix allows for a fast look-up of free text associated with a list of identifiers, when a searchIndex has also been created. See a Searchable Track Hub Quick Start Guide here. 9. Extra fields can contain text for labels or for display with mouseover (if the BED "name" field is needed for something that is not the label). See the trackDb settings "mouseOverField" and "labelField" for more information. 10. When you click on features, the contents of all extra fields are shown as a table. You can modify the layout of the resulting page with the trackDb settings "skipFields", "sepFields" and "skipEmptyFields", and transform text fields into links with the "urls" trackDb setting. 11. Extra fields that start with the character "_" are reserved for internal use (special display code); their contents are not shown on the details page. Sharing Your Data with Others If you would like to share your bigBed data track with a colleague, the best solution is to save your current view as a stable Genome Browser Session Link. This will save the position and all settings that you made, all track visibilities, filters, highlights, etc. If you want to create URLs to your bigBed file programmatically from software, look at Example #6 on this page. Extracting Data from the bigBed Format Because the bigBed files are indexed binary files, it can be difficult to extract data from them. UCSC has developed the following programs to assist in working with bigBed formats, available from the binary utilities directory: - bigBedToBed — converts a bigBed file to ASCII BED format. - bigBedSummary — extracts summary information from a bigBed file. - bigBedInfo — prints out information about a bigBed file. These programs accept either file names or URLs to files as input. As with all UCSC Genome Browser programs, simply type the program name (with no parameters) on the command line to view the usage statement. Troubleshooting If you get an error when you run the bedToBigBed program, check your input BED file for data coordinates that extend past the end of the chromosome. If these are present, run the bedClip program (available here) to remove the problematic row(s) in your input BED file before using the bedToBigBed program. 
/goldenPath/help/assemblyHubGuidelines.html:Public_Hub_Guidelines	Assembly Please note, if you are working with a genome that has already been submitted to the NCBI Assembly system, it may already be available in the UCSC Genome Browser. Please examine the GenArk Assembly Hub collection to see if your genome of interest is already available. In the case it cannot be found there, you can use the UCSC Assembly Request page to request a genome assembly be added to the UCSC Genome Browser. Contents Overview Web Server Linking to Your Assembly Hub - hub.txt - genomes.txt - 2bit File - groups.txt Building Tracks - Cyotoband Track Assembly Hub Resources - G-OnRamp - MakeHub - Example NCBI Assembly Hubs - Example Loading African Bush Elephant Assembly Hub and Looking at the Related genomes.txt and trackDb.txt Adding BLAT Servers - Configuring Assembly Hubs to Use a Dedicated gfServer - Troubleshooting BLAT Servers - Process Check - Check for Correct Path/Filename - Check "gfServer Status" Check - Testing with gfClient - Configuring Assembly Hubs to Use a Dynamic gfServer - Check gfServer Status for Dynamic Servers Overview The Assembly Hub function allows you to display your novel genome sequence using the UCSC Genome Browser. Web Server To display your novel genome sequence, use a web server at your institution (or free services like Cyverse), for usage behind a firewall you can also load them locally through GBiB to supply your files to the UCSC Genome Browser. Note that hosting hub files on HTTP is highly recommended and much more efficient than FTP. You then establish a hierarchy of directories and files to host your novel genome sequence. For example: myHub/ - directory to organize your files on this hub hub.txt - primary reference text file to define the hub, refers to: genomes.txt - definitions for each genome assembly on this hub newOrg1/ - directory of files for this specific genome assembly newOrg1.2bit - ‘2bit’ file constructed from your fasta sequence description.html - information about this assembly for users trackDb.txt - definitions for tracks on this genome assembly groups.txt - definitions for track groups on this assembly bigWig and bigBed files - data for tracks on this assembly external track hub data tracks The URL to reference this hub would be: http://yourLab.yourInstitution.edu/myHub/hub.txt Note: there is now a useOneFile on hub setting that allows the hub properties to be specified in a single file. More information about this setting can be found on the Genome Browser User Guide. You can view a working example hierarchy of files at: Plants A smaller slice of this hub is represented in a Quick Start Guide to Assembly Hubs. Linking to Your Assembly Hub You can build direct links to the genome(s) in your assembly hub: - The hub connect page: http://genome.ucsc.edu/cgi-bin/hgHubConnect?hgHub_do_redirect=on&hgHubConnect.remakeTrackHub=on&hgHub_do_firstDb=1&hubUrl=http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubAssembly/plantAraTha1/hub.txt - The genome gateway page: http://genome.ucsc.edu/cgi-bin/hgGateway?genome=araTha1&hubUrl=http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubAssembly/plantAraTha1/hub.txt - Directly to the genome browser: http://genome.ucsc.edu/cgi-bin/hgTracks?genome=araTha1&hubUrl=http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubAssembly/plantAraTha1/hub.txt hub.txt The initial file hub.txt is the primary URL reference for your assembly hub. The format of the file: hub hubName shortLabel genome longLabel Comment describing this hub contents genomesFile genomes.txt email contactEmail@institution.edu descriptionUrl aboutHub.html shortLabel is the name that will appear in the genome pull-down menu at the UCSC gateway page. Example: Plants. genomesFile is a reference to the next definition file in this chain that will describe the assemblies and tracks available at this hub. Typically genomes.txt is at the same directory level as this hub.txt, however it can also be a relative path reference to a different directory level. The email address provides users a contact point for queries related to this assembly hub. The descriptionUrl provides a relative path or URL link to a webpage describing the overall hub. genomes.txt The genomes.txt file provides the references to the genome assemblies and tracks available at this assembly hub. The example file indicates the typical contents: genome ricCom1 trackDb ricCom1/trackDb.txt groups ricCom1/groups.txt description July 2011 Castor bean twoBitPath ricCom1/ricCom1.2bit organism Ricinus communis defaultPos E09R7372:1000000-2000000 orderKey 4800 scientificName Ricinus communis htmlPath ricCom1/description.html transBlat yourLab.yourInstitution.edu 17777 blat yourLab.yourInstitution.edu 17777 isPcr yourLab.yourInstitution.edu 17779 There can be multiple assembly definitions in this single file. Separate these stanzas with blank lines. The references to other files are relative path references. In this example there is a sub-directory here called ricCom1 which contains the files for this specific assembly. - The genome name is the equivalent to the UCSC database name. The genome browser displays this database name in title pages in the genome browser. - The trackDb refers to a file which defines the tracks to place on this genome assembly. The format of this file is described in the Track Hub help reference documentation. - The groups refers to a file which defines the track groups on this genome browser. Track groups are the sections of related tracks grouped together under the primary genome browser graphics display image. - The description will be displayed for user information on the gateway page and most title pages of this genome assembly browser. It is the name displayed in the assembly pull-down menu on the browser gateway page. - The twoBitPath refers to the .2bit file containing the sequence for this assembly. Typically this file is constructed from the original fasta files for the sequence using the kent program faToTwoBit. This line can also point to a URL, for example, if you are duplicating an existing Assembly Hub, you can use the original hub's 2bit file's URL location here. - The organism string is displayed along with the description on most title pages in the genome browser. Adjust your names in organism and description until they are appropriate. This example is very close to what the genome browser normally displays. This organism name is the name that appears in the genome pull-down menu on the browser gateway page. - The defaultPos specifies the default position the genome browser will open when a user first views this assembly. This is usually selected to highlight a popular gene or region of interest in the genome assembly. - The orderKey is used with other genome definitions at this hub to order the pull-down menu ordering the genome pull-down menu. - The htmlPath refers to an html file that is used on the gateway page to display information about the assembly. - The transBlat, blat, and isPcr entries refer to different configurations of the gfServer that enhance search capabilities for amino acids, BLAT algorithms, and PCR respectively. More here. Note that it is strongly encouraged to give each of your genomes stanza's a line for defaultPos, scientificName, organism, description (along with other above settings) so that when your hub is attached it will load a specified default location and have text to be more easily searched from the Gateway page. 2bit File The .2bit file is constructed from the fasta sequence for the assembly. The kent source program faToTwoBit is used to construct this file. Download the program from the downloads section of the Browser. For example: faToTwoBit ricCom1.fa ricCom1.2bit Use the twoBitInfo to verify the sequences in this assembly and create a chrom.sizes file which is not used in the hub, but is useful in later processing to construct the big* files: twoBitInfo ricCom1.2bit stdout | sort -k2rn > ricCom1.chrom.sizes The .2bit commands can function with the .2bit file at a URL: twoBitInfo -udcDir=http://genome-test.gi.ucsc.edu/~hiram/hubs/Plants/ricCom1/ricCom1.2bit stdout | sort -k2nr > ricCom1.chrom.sizes Sequence can be extracted from the .2bit file with the twoBitToFa command, for example: twoBitToFa -seq=chrCp -udcDir=http://genome-test.gi.ucsc.edu/~hiram/hubs/Plants/ricCom1/ricCom1.2bit stdout > ricCom1.chrCp.fa groups.txt The groups.txt file defines the grouping of track controls under the primary genome browser image display. The example referenced here has the usual definitions as found in the UCSC Genome Browser. Each group is defined, for example the Mapping group: name map label Mapping priority 2 defaultIsClosed 0 - The name is used in the trackDb.txt track definition group, to assign a particular track to this group. - The label is displayed on the genome browser as the title of this group of track controls. - The priority orders this track group with the other track groups. - The defaultIsClosed determines if this track group is expanded or closed by default. Values to use are 0 or 1. Building Tracks Tracks are defined in the trackDb.txt where each stanza describes how tracks are displayed (shortLabel/longLabel/color/visibility) and other information such as what group the track should belong to (referencing the groups.txt) and if any additional html should display when one clicks into the track or a track item: track gap_ longLabel Gap shortLabel Gap priority 11 visibility dense color 0,0,0 bigDataUrl bbi/ricCom1.gap.bb type bigBed 4 group map html ../trackDescriptions/gap For more informations about the syntax of the trackDb.txt file, use UCSC's Hub Track Database Definition page. It helps to have a cluster super computer to process the genomes to construct tracks. It can be done for small genomes on single computers that have multiple cores. The process for each track is unique. Please note the continuing document: Browser Track Construction for a discussion of constructing tracks for your assembly hub. Cytoband Track Assembly hubs can have a Cytoband track that can allow for quicker navigation of individual chromosomes and display banding pattern information if known. A quick version of the track can be built using the existing chrom.sizes files for your assembly (the banding options include gneg, gpos25, gpos50, gpos75, gpos100, acen, gvar, or stalk): cat araTha1.chrom.sizes | sort -k1,1 -k2,2n | awk '{print $1,0,$2,$1,"gneg"}' > cytoBandIdeo.bed The resulting bed file can be turned into a big bed and given a .as file (example here) to inform the browser it is not a normal bed. bedToBigBed -type=bed4 cytoBandIdeo.bed -as=cytoBand.as araTha1.chrom.sizes cytoBandIdeo.bigBed In the trackDb, as long as the track is named cytoBandIdeo (track cytoBandIdeo example) it will load in the assembly hub. Assembly Hub Resources There are resources for automatically building assembly hubs available from G-OnRamp and MakeHub. There is also a collection of Example NCBI assembly hubs that are already working and can either be used or copied as a template to build further hubs. G-OnRamp G-OnRamp is a Galaxy workflow that turns a genome assembly and RNA-Seq data into a Genome Browser with multiple evidence tracks. Because G-OnRamp is based on the Galaxy platform, developing some familiarity with the key concepts and functionalities of Galaxy would be beneficial prior to using G-OnRamp. Here is a link to their instruction page that gives an overview of their process. MakeHub MakeHub is a command line tool for the fully automatic generation of track data hubs for visualizing genomes with the UCSC genome browser. More information can be found on their GitHub page. Example loading African bush elephant assembly hub and looking at the related genomes.txt and trackDb.txt Here are some quick steps to load an example hub from this collection, and an attempt to explain how to look at the files behind the hub. 1. Click the above Vertebrate Mammalian assembly hub link. 2. Scroll down and find the "common name" column and click the hyperlink for "African bush elephant" after looking at the other information on that row. 3. Note that you have arrived at a gateway page that has "African bush elephant Genome Browser - GCA_000001905.1_Loxafr3.0" displayed, where you can see a "Download files for this assembly hub:" section if you desired to access these specific files and notably a link. 4. Click "Go" or the top "Genome Browser" blue bar menu to arrive at viewing this assembly hub (note this is on our genome-test site). 5. To load this hub on our public site, at the earlier step you can copy the hyperlink for "African bush elephant" and paste it in a browser and change the very first "http://genome-test.gi.ucsc.edu/gbdb/..." to "http://genome.ucsc.edu/cgi-bin/..." instead. Now to investigate the files behind the hub to understand the process involved: 1. Click the link found in the "Download files for this assembly hub:" section on a loaded assembly hub's gateway page. 2. Note the "GCA_000001905.1_Loxafr3.0.ncbi2bit" file, this is the binary indexed remote file that is allowing the Browser to display this genome. 3. Find the "GCA_000001905.1_Loxafr3.0.genomes.ncbi.txt" file and click the link to look at it. 4. Review this genomes.txt file, which defines each track in a new hub to show where to find the above 2bit on the "twoBitPath" line and also defines where to find all track database to display data on this genome in the "trackDb" line (the real genomes.txt for this massive hub is up one directory as this hub has 204 assemblies - where you will find this stanza included). 5. From the earlier link to all the files, click the GCA_000001905.1_Loxafr3.0.trackDb.ncbi.txt link. 6. Review this trackDb.txt file which defines the tracks to display on this hub, and also has "bigDataUrl" lines to tell the Browser where to find the data to display for each track, as well as other features such on some tracks as "searchIndex" and "searchTrix" lines to help support finding data in the hub and "url" and "urlLabel" lines on some tracks to help create links out on items in the hub to other external resources and "html" lines to a file that will have information to display about the data for users who click into tracks. Adding BLAT servers BLAT servers (gfServer) are configured as either dedicated or dynamic servers. Dedicated BLAT serves index a genome when started and remain running in memory to quickly respond to request. Dynamic BLAT servers pre-index genomes to files and are run on demand to handle a BLAT request and then exit. Dedicated gfServer are easier to configure and faster to respond. However, the server continually uses memory. A dynamic gfServer is more appropriate with multiple assemblies and infrequent use. Their response time is usually acceptable; however, it varies with the speed of the disk containing the index. With repeated access, the operating system will cache the indexes in memory, improving response time. Configuring assembly hubs to use a dedicated gfServer By running your own BLAT server, you can add lines to the genomes.txt file of your assembly hub to enable the browser to access the server and activate blat searches. Please see Running your own gfServer for details on installing and configuring both dedicated and dynamic gfServers. - Next edit your genomes.txt stanza that references yourAssembly to have two lines to inform the browser of where the blat servers are located and what ports to use. See an example of commented out lines here. Please note the capital "B" in transBlat. transBlat yourServer.yourInstitution.edu 17777 blat yourServer.yourInstitution.edu 17779 isPcr yourServer.yourInstitution.edu 17779 - You should now be able to load and perform blat and PCR operations on your assembly. For example, a URL such as the following would bring up the blat CGI and have your assembly listed at the bottom of the "Genome:" drop-down menu: http://genome.ucsc.edu/cgi-bin/hgBlat?hubUrl=http://yourServer.yourInstitution.edu/myHub/hub.txt. Also note the separate isPcr line provides the option to use a different gfServer than the blat host if desired. - Some institutions have firewalls that will prevent the browser from sending multiple inquiries to your blat servers, in which case you may need to request your admins add this IP range as exceptions that are not limited: 128.114.119.* That will cover the U.S. genome.ucsc.edu site. In case you may wish the requests to work from our European Mirror genome-euro.ucsc.edu site, you would want to include 129.70.40.120 also to the exception list. Please see more about configuring your blat gfServer to replicate the UCSC Browser's settings, which will also have information about optimizing PCR results. The Source Downloads page offers access to utilities with pre-compiled binaries such as gfserver found in a blat/ directory for your machine type here and further blat documentation here, and the gfServer usage statement for further options. Please also know you can set up gfservers on a GBiB and run it locally. Please see this GBiB assembly blat step-by-step set up page for details. Note: You can stop your instance of gfServer with a command. For example: gfServer stop localhost 17860 Troubleshooting BLAT servers You can see this error if you have the translatedBlat / nucleotideBlat port numbers the wrong way around: Expecting 6 words from server got 2 The following is an example of an error message when attempting to run a DNA sequence query via the web-based BLAT tool after loading a hub, after starting a gfServer instance (from the same dir as the 2bit file). For example, a command to start an instance of gfServer: gfServer start localhost 17779 -stepSize=5 contigsRenamed.2bit & Example of a possible error message, from web-based BLAT after attempting a web-based BLAT query: Error in TCP non-blocking connect() 111 - Connection refused Operation now in progress Sorry, the BLAT/iPCR server seems to be down. Please try again later. Check the following: 1.) Process check First, make sure your gfServer instance is running. Type the following command to check for your running gfServer process: ps aux | grep gfServer 2.) Check for correct path/filename In your genomes.txt file, does your twoBitPath/filename match what you specified in your command to start gfServer? In your genomes.txt file, is the location of the instance to your gfServer correct? To check this, you can cd into the directory where you started your gfServer, then type the command: hostname -i Your result should be an IP address, for example, '132.249.245.79'. Now you can test the connection to your port that you specified, with a simple telnet command. Type in the following command: telnet yourIP yourPort. For example: telnet 132.249.245.79 17777 The results should read, "Connected to 132.249.245.79". Otherwise, if gfServer isn't running or if you typed the wrong location in your telnet command, telnet will say, "Connection refused." In this example, check your genomes.txt file, and make sure your blat line reads, "blat 132.249.245.79 17777". You may need to change your genomes.txt file from, for example, "blat localhost 17777" to "blat 132.249.245.79 17777" (use your specific IP/host name where gfServer is running). 3.) Check "gfServer status" check To request status from the gfServer process, run: gfServer status yourLocation yourPort. For example: $ gfServer status 132.249.245.79 17777 You should see output like this: version 36x2 type nucleotide host localhost port 17777 tileSize 11 stepSize 5 minMatch 2 pcr requests 0 blat requests 0 bases 0 misses 0 noSig 1 trimmed 0 warnings 0 4.) Testing with gfClient The best troubleshooting test is to take the webpage out of the equation, and use the command line utility, gfClient, to run the query on your instance of gfServer. If you can successfully connect gfClient to gfServer, you will know that your location and port specification are correct. From the directory that holds your hub's .2bit file (should be the same directory where your instance of gfServer was launched), perform a query using gfClient: You can type "gfClient" on your command line to see the usage statement. Use the following command: gfClient yourLocation yourPort pathOf2bitFile yourFastaQuery.fa nameOfOutputFile.psl FYI: For testing with gfClient, you only need the gfServer binary on your server, not blat. For example: gfClient localhost 17777 . query.fa gfOutput.psl Note the "." after the port, to specify that the query will use the .2bit file in the current directory. After running this command, take a look at the gfOutput.psl file. If successful, you will see BLAT results. Another example: Note: In the example below, "yourServer.yourInstitution.edu" is the name of their machine where you run the gfServer command. From the test machine: Test the DNA alignment, where test.fa is some sequence to find: gfClient yourServer.yourInstitution.edu 17779 `pwd` test.fa dnaTestOut.psl From the test machine: Test the protein alignment, where proteinSequence.fa is the sequence to find: gfClient -t=dnaX -q=prot yourServer.yourInstitution.edu 17779 `pwd` proteinSequence.fa proteinOut.psl - NOTE: the yourAssembly.2bit file needs to be on this test machine also. - The pwd says to find the yourAssembly.2bit file in this directory. Configuring assembly hubs to use a dynamic gfServer A dynamic BLAT server is specified with the "dynamic" argument to the blat, transBlat, isPcr definitions in the hub genomes.txt file, followed by the gfServer root-relative path of the directory containing the 2bit and gfidx files. For example: blat yourServer.yourInstitution.edu 4096 dynamic yourAssembly transBlat yourServer.yourInstitution.edu 4096 dynamic yourAssembly isPcr yourServer.yourInstitution.edu 4096 dynamic yourAssembly The genome and gfServer indexes would be: $rootdir/yourAssembly/yourAssembly.2bit $rootdir/yourAssembly/yourAssembly.untrans.gfidx $rootdir/yourAssembly/yourAssembly.trans.gfidx See Building gfServer indexes for instructions in building the index. For large hubs, it is possible to have more deeply nest directory, for instance, the following NCBI convention: blat yourServer.yourInstitution.edu 4096 dynamic GCF/000/181/335/GCF_000181335.3 transBlat yourServer.yourInstitution.edu 4096 dynamic GCF/000/181/335/GCF_000181335.3 isPcr yourServer.yourInstitution.edu 4096 dynamic GCF/000/181/335/GCF_000181335.3 Which will reference these genome files and indexes: $rootdir/GCF/000/181/335/GCF_000181335.3/GCF_000181335.3.2bit $rootdir/GCF/000/181/335/GCF_000181335.3/GCF_000181335.3.untrans.gfidx $rootdir/GCF/000/181/335/GCF_000181335.3/GCF_000181335.3.trans.gfidx Check gfServer status for dynamic servers A query without specifying a genome is an "I am alive" check: % gfServer status myserver 4040 version 37x1 serverType dynamic Specifying a genome checks that is is valid and gives information on how to the index was built: % gfServer -genome=mm10 -genomeDataDir=test/mm10 status myserver 4040 version 37x1 serverType dynamic type nucleotide tileSize 11 stepSize 5 minMatch 2 Using -trans checks the translated index: % gfServer -genome=mm10 -genomeDataDir=test/mm10 -trans status myserver 4040 version 37x1 serverType dynamic type translated tileSize 4 stepSize 4 minMatch 3 
/goldenPath/help/customTrack.html:Genome_Browser_Custom_Tracks	Displaying Your Own Annotations in the Genome Browser Table of Contents What are custom annotation tracks? - Building and sharing a custom track Loading a custom track into the Genome Browser Displaying and managing custom tracks - Creating browser lines for annotations - Additional browser line options - Defining track lines for annotations - Required and useful track attribute pairs Sharing your annotation track with others Troubleshooting annotation display problems Custom Track Examples: - Simple annotation file - Two annotations track in one file - BED custom track with multiple blocks - Simple annotation in bigBed format - Create external links using the 'name' field from the BED file - Loading a custom track via the URL - Construct a sharable URL using the bigDataUrl setting 
/goldenPath/help/query.html:Genome_Browser_Queries	Querying the Genome Browser From the Genomes page, you can jump to the default position of an assembly by clicking the "Go" button or you can specify a particular genome position in a variety of formats. These same formats are valid in the search bar above the main Genome Browser track display. In addition to the positional queries described below, any search term can be used to find matches in track data, track names and/or descriptions, help docs, and public hub track names and/or descriptions. See our search page for more details on the search functionality. Valid position queries can include: - Chromosome numbers - Chromosomal coordinate ranges - Gene names - Accession numbers - An mRNA, EST or STS marker - Keywords from the GenBank description of an mRNA - HGVS terms - HGVS and accession searches on outdated RefSeq accession versions is available on hg38 To specify a genome position: 1. Select the desired clade, genome and assembly 2. Enter the desired query in the "Position/Search Term" box (see sample queries below) 3. Click the "Go" button A query may have multiple results. If this is the case, a results page will appear listing each result along with the track it is associated with. Once selected, the result will be displayed in the Browser with a highlighted label, making it easier to identify. If you have further questions, you can search the Genome Browser FAQ page and find links to further resources. Also, developers of track hubs can create searchable track hubs using the searchTrix setting. To quickly jump to a codon or exon of a gene transcript: 1. Use one of the searches below to jump to a gene, to show all transcripts of a gene or range of interest 2. Right-click any transcript, select "Choose exon" or "Zoom to codon" and enter the exon or codon position of interest Sample queries Below is a list of examples that might be used to query the Genome Browser. Note that not every query listed here will produce a result in every assembly. The list serves only to illustrate the different types of queries that can be performed. Query Genome Browser Response chr7 Displays all of chromosome 7 chr3:1-1000000 Displays the first million bases of chromosome 3, counting from the p-arm telomere 3:1-1000000 Displays the first million bases of chromosome 3, Ensembl format chromosome names chr3 0 1000000 Displays the first million bases of chromosome 3; BED format NC_000007.14:1-1000000 Displays the first million bases of chromosome 3, RefSeq format CM000665.2:1-1000000 Displays the first million bases of chromosome 3, GenBank/INSDC format chr3:1000000+2000 Displays a region of chromosome 3 that spans 2000 bases, starting with position 1000000 chrUn_GL000213v1 Displays all of the unplaced contig GL000213v1 chr3_GL000221v1_random Displays the unlocalized contig GL000221v1 chr1_KN196472v1_fix Displays all of patch fix KN196472v1 20p13 Displays the region for band p13 on chromosome 20 GTATGTAGCCACGGAGCACCATTACCTGTCACCATTACCTGAATGGCTA Displays the first best match to this DNA sequence, e.g. chr21:33034835-33034883 for hg19 AA205474 Displays the region containing the EST with GenBank accession AA205474 in the BRCA1 cancer gene on chromosome 17 AC008101 Displays the region containing the clone with GenBank accession AC008101 AF083811 Displays the region containing the mRNA with GenBank accession number AF083811 NM_017414 Displays the region containing RefSeq identifier NM_017414 NP_059110 Displays the region containing protein accession number NP_059110 PRNP Displays the region containing HUGO Gene Nomenclature Committee identifier PRNP Q99697 Displays the region containing the alignment of the UniProt/SwissProt protein sequence with accession Q99697 (PITX2) RH18061;RH80175 15q11;15q13 NM_012090.5;NM_012421.4 Displays the region between genome landmarks, such as the STS markers RH18061 and RH80175, or chromosome bands 15q11 to 15q13, or SNPs NM_000310.4 and NM_012090.5. This syntax may also be used for other range queries, such as between uniquely determined ESTs, mRNAs, refSeqs, SNPS, etc. NR_026861.1:1-1000 Works with any other type of accession from this page: Displays the first 1000bp of NR_026861.1 NM_000310.4(PPT1):c.271_287del17insTT NM_007262.5(PARK7):c.-24+75_-24+92dup NM_006172.4(NPPA):c.456_*1delAA MYH11:c.503-14_503-12del NM_198576.4(AGRN):c.1057C>T NM_198056.3:c.1654G>T NP_002993.1:p.Asp92Glu NP_002993.1:p.D92E BRCA1 Ala744Cys BRCA1 A744C LRG_100t1:c.4G>A LRG_100t1:n.1 LRG_456p1:p.Ser190Leu LRG_321:g.16409_16461del ENST00000002596.6:c.-108-6848A>G ENSP00000005178.5:p.Val20Gly chrX:g.31500000_31600000del NR_111987:n.-1 NM_015102.5:n.3038-2 NM_001372044:c.1528_1530del Displays the region that matches the HGVS expression, usually in the format <transcript or protein>:<position> <amino acid or nucleotide change> If a gene symbol is used, HGVS search will try all RefSeq transcripts to find the nucleotide or amino acid at the position indicated in the expression. If there are multiple matches, a disambiguation page will be shown. If the RefSeq sequence differs from the genome sequence, then currently the search will use the genome, not the transcript, for codon counting and amino acid / nucleotide comparison. Please contact us if this is inconvenient. NM_198056.2:c.1A>C An example of an HGVS search on a previous NM version that is now outdated. Support for previous NM accessions is only available on hg38. essv8694097 Displays the region covering the copy number variant with the accession essv8694097 in the Database of Genomic Variants (DGV) nssv3446126 Displays the region covering the copy number variant with the accession nssv3446126 in the cases of developmental delay CTD-3071L10 Displays the region covering the CTD-3071L10 NCBI clone end mapping in the NCBI Clone DB database nssv16167444 Displays the region covering the common copy number genomic variant with the accession nssv16167444 in the nstd186 (NCBI Curated Common Structural Variants) dataset rs1333049 Displays results for annotations matching this rsID, including dbSNP database COSM6161404 Displays the region covering COSM6161404 in the Catalogue Of Somatic Mutations In Cancer (COSMIC) database nssv3395351 Displays the region covering ClinVar Copy Number Variant with the accession nssv3395351 in the ClinVar database BRCT_assoc Displays the region covering the manually-curated Pfam-A domain BRCT_assoc found in GENCODE Genes U133A:219211_at Displays the region containing the consensus and exemplar sequences used for the selection of probes on the Affymetrix HG-U133A chips chr1 0 1000 When entered without ":" and "-", uses 0-based, half-open coordinates (like custom tracks and internal table coordinates), so displays chr1:1-1000 pseudogene mRNA Lists transcribed pseudogenes, but not cDNAs p53 Lists mRNAs related to the p53 tumor suppressor T-cell receptor Lists mRNAs for T-cell receptor genes in GenBank breast cancer Lists mRNAs associated with breast cancer homeobox caudal Lists mRNAs for caudal homeobox genes zinc finger Lists zinc finger mRNAs kruppel zinc finger Lists only kruppel-like zinc fingers huntington Lists candidate genes associated with Huntington's disease zahler Lists mRNAs deposited by a scientist named Zahler Evans,J.E. Lists mRNAs deposited by co-author J.E. Evans Use this last format for author queries. Although GenBank requires the search format Evans JE, internally it uses the format Evans,J.E.. 
/goldenPath/help/bigLolly.html:Genome_Browser_bigLolly_Track_Format	bigLolly Track Format The bigLolly format uses a standard bigBed file that is used to generate a lollipop graph where the position of a lollipop circle corresponds to a genomic coordinate. By default, the score is used to decide how high to draw the lollipop, but there are trackDb options to specify which fields to use for the height and width of the lollipop, as well as to draw lines on the graph. BigLolly trackDb options arguments are noStems, lollySizeField, lollyMaxSize, lollyField, yAxisLabel, and yAxisNumLabels. These options are also described in the trackDb help doc. This format is useful for displaying small genomic features such as sequence variants, as it provides two ways to characterize features and make them more visible -- stem height and radius -- in addition to color. The lollipop graph type can be used to annotate bases for variants, RNA editing, Selenocysteines, frameshifts, or any other reason. [] The bigBed files used in bigLolly type are in an indexed binary format. The main advantage of this format is that only those portions of the file needed to display a particular region are transferred to the Genome Browser server. The bigLolly file remains on your local web-accessible server (http, https or ftp), not on the UCSC server, and only the portion needed for the currently displayed chromosomal position is locally cached as a "sparse file". If you do not have access to a web-accessible server and need hosting space for your bigLolly files, please see the Hosting section of the Track Hub Help documentation. Contents bigLolly format definition Creating a bigLolly track Sharing your data with others Extracting data from the bigLolly format Troubleshooting bigLolly format definition Any bigBed file can be displayed as a bigLolly. See bigBed format. The following autoSql definition is an example on how to specify bigLolly files. This definition, contained in the file bigLolly.as, is pulled in when the bedToBigBed utility is run with the -as=bigLolly.as option. table bigLolly "bigLolly lollipops" ( string chrom; "Reference sequence chromosome or scaffold" uint chromStart; "Start position in chrom" uint chromEnd; "End position in chrom" string name; "dbSNP Reference SNP (rs) identifier or :" uint score; "Score from 0-1000, derived from p-value" char[1] strand; "Unused. Always '.'" uint thickStart; "Start position in chrom" uint thickEnd; "End position in chrom" uint color; "Red (positive effect) or blue (negative). Brightness reflects pvalue" double pValueLog; "-log10 p-value" ) The first 9 fields of this bigLolly format are the same as the first 9 fields of the standard BED format. The pValueLog field provides a numeric field for stem height. Creating a bigLolly track Example #1 In this example, you will create a bigLolly custom track using an existing bigBed file, located on the UCSC Genome Browser http server. By default the score field is used to define the lollipop height. This file contains data for the hg38 assembly. To create a custom track using this bigBed file: 1. Construct a track line that references the file: track type=bigLolly name="bigLolly Example One" description="A bigLolly file" bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigBedExample3.bb visibility=full 2. Paste the track line into the custom track management page for the human assembly hg38 (Dec. 2013). 3. Click the "submit" button. 4. Go to chr21:17,030,007-17,055,589 to see the data. Example #2 In this example, you will create your own bigBed file to display as a bigLolly from an bed file, using an extra field to define the height of the lollipops. 1. Save this bed file to your computer. 2. Save the autoSql files bigLollyExample2.as to your computer. 3. Download the bedToBigBed utility. 4. Save the hg38.chrom.sizes text file to your computer. This file contains the chrom.sizes for the human hg38 assembly. 5. Use the bedToBigBed utility to create a bigBed file from your sorted BED file, using the bigLollyExample2.bed file and chrom.sizes files created above. bedToBigBed -as=bigLollyExample2.as -type=bed9+1 bigLollyExample2.bed hg38.chrom.sizes bigLollyExample2.bb 6. Move the newly created bigBed file (bigLollyExample2.bb) to a web-accessible http, https, or ftp location. At this point you should have a URL to your data, such as "https://institution.edu/bigLollyExample2.bb", and the file should be accessible outside of your institution/hosting providers network. For more information on where to host your data, please see the Hosting section of the Track Hub Help documentation. track type=bigLolly name="bigLolly Example Two: SNP data" description="A second bigLolly file" bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigLollyExample2.bb lollyField=pValueLog visibility=full 7. Go to chr21:15,593,670-15,632,442 to see the data. Example #3 In this example, you will create your own bigBed file to display as a bigLolly from a bed file with the size of the lollipop defined by an extra field (lollySizeField=lollySize) where the numbers in this field are similar to a radius and define circle size. To avoid large circles from being clipped, the setting lollyMaxSize=10 ensures circles of size 10 fully display. Also, to turn off the lollipop stems, the setting lollyNoStems=on is added. Finally, the settings yAxisLabel.0="0 on 30,30,190 0" and yAxisLabel.1="5 on 30,30,190 5" adds labels and lines on the y axis where 30,30,190 defines the color. 1. Save this bed file to your computer. 2. Save the autoSql files bigLollyExample3.as to your computer. 3. Download the bedToBigBed utility. 4. Save the hg38.chrom.sizes text file to your computer. This file contains the chrom.sizes for the human hg38 assembly. 5. Use the bedToBigBed utility to create a bigBed file from your sorted BED file, using the bigLollyExample3.bed file and chrom.sizes files created above. bedToBigBed -as=bigLollyExample3.as -type=bed9+1 bigLollyExample3.bed hg38.chrom.sizes bigLollyExample3.bb 6. Move the newly created bigBed file (bigLollyExample3.bb) to a web-accessible http, https, or ftp location. At this point you should have a URL to your data, such as "https://institution.edu/bigLollyExample3.bb", and the file should be accessible outside of your institution/hosting providers network. For more information on where to host your data, please see the Hosting section of the Track Hub Help documentation. track type=bigLolly name="bigLolly Example Three" description="A third bigLolly file" bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigLollyExample3.bb lollySizeField=lollySize visibility=full yAxisLabel.0="0 on 30,30,190 0" yAxisLabel.1="5 on 30,30,190 5" lollyMaxSize=10 lollyNoStems=on 7. Go to chr21:25,891,755-25,891,870 to see the data. [] Sharing your data with others Custom tracks can also be loaded via one URL line. This link loads the same bigLolly.bb track and sets additional display parameters from Example 1 in the URL: http://genome.ucsc.edu/cgi-bin/hgTracks?ignoreCookie=1&db=hg38&position=chr21:17,002,145-17,159,243&hgct_customText=track%20type=bigLolly%20name=Example %20bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigBedExample3.bb %20visibility=full If you would like to share your bigLolly data track with a colleague, learn how to create a URL link to your data by looking at Example #6 on the custom track help page. Extracting data from the bigLolly format Because the bigLolly files are an extension of bigBed files, which are indexed binary files, it can be difficult to extract data from them. UCSC has developed the following programs to assist in working with bigBed formats, available from the binary utilities directory. - bigBedToBed — converts a bigBed file to ASCII BED format. - bigBedSummary — extracts summary information from a bigBed file. - bigBedInfo — prints out information about a bigBed file. As with all UCSC Genome Browser programs, simply type the program name (with no parameters) at the command line to view the usage statement. Troubleshooting If you encounter an error when you run the bedToBigBed program, check your input file for data coordinates that extend past the the end of the chromosome. If these are present, run the bedClip program (available here) to remove the problematic row(s) in your input file before running the bedToBigBed program. 
/goldenPath/help/hgTablesHelp.html:Table_Browser_Help	Table Browser User's Guide Contents Introduction About the Table Browser databases and tables - Position-oriented tables - Non-positional tables Getting started - simple queries - Simple position-based query - Batch query using identifiers - Batch query using positions - Query to get gene symbols Filtering output by constraining field values - Filtering on fields from a single table - Filtering on fields from multiple tables - Filter constraints Intersecting data from multiple tables - Intersecting data from two tables - Intersecting data from multiple tables - Intersection options Correlating data from two tables Output formats - Displaying all fields in a table - Displaying selected fields from one or more tables - Displaying sequence (FASTA) data - Displaying CDS FASTA alignments - Saving query results in GTF or BED format - Saving data to a file - Saving data as a custom track - Displaying query results as Genome Browser hyperlinks - Displaying a statistical summary of query data Video examples of Table Browser queries - Find list of genes in a region - Obtaining coordinates and sequences of gene exons - Find SNPs in a gene - Find SNPs upstream of Genes ------------------------------------------------------------------------ Search the Genome Browser help pages:   Questions and feedback are welcome. Introduction The Table Browser provides a powerful and flexible graphical interface for querying and manipulating the Genome Browser annotation tables. Because the Table Browser uses the same database as the Genome Browser, the two views are always consistent. Using the Table Browser, you can: - retrieve the DNA sequence data or annotation data underlying Genome Browser tracks for the entire genome, a specified coordinate range, or a set of accessions - apply a filter to set constraints on field values included in the output - generate a custom track and automatically add it to your session so that it can be graphically displayed in the Genome Browser - conduct both structured and free-from SQL queries on the data - combine queries on multiple tables or custom tracks through an intersection or union and generate a single set of output data - display basic statistics calculated over a selected data set - display the schema for table and list all other tables in the database connected to the table - organize the output data into several different formats for use in other applications, spreadsheets, or databases This User's Guide is aimed at both the novice Table Browser user as well the advanced user. If you are new to the Table Browser, read the Getting started section to learn about browser basics and try some simple queries. Advanced users may want to proceed directly to the section that addresses a particular area of functionality in detail. Although the Table Browser provides sufficient flexibility to satisfy the needs of most users, some advanced users may require the ability to run SQL commands directly on the Genome Browser database. UCSC provides two public MariaDB servers: (1) genome-mysql.soe.ucsc.edu (US West Coast), (2) genome-euro-mysql.soe.ucsc.edu (Europe). More information can be found on our MariaDB Access page. Alternatively, the database may be downloaded to a local computer for MariaDB access. See the mirror site documentation for information on setting up a local copy of the database. About the Table Browser databases and tables The Table Browser is built on top of the Genome Browser database, which actually consists of several separate databases, one for each genome assembly. Tables within the databases may be differentiated by whether the data are based on genome start-stop coordinates (positional tables) or are independent of position (non-positional tables).Some output formats and query options are applicable only to positional tables, hence the distinction. Positional tables Positional tables contain data associated with specific locations in the genome, such as mRNA alignments, gene predictions, cross-species alignments, and other annotations. Each of the annotation tracks displayed in the Genome Browser is based on a positional table. In some instances, data from other positional and non-positional tables may also be incorporated into the track. Data associated with custom annotation tracks active within the user's Table Browser session are also available as positional tables. Positional tables can be further subdivided into several categories based on the type of data they describe. Alignment data can be best described by using a block structure to represent each element. Other tables require only start and end coordinate data for each element. Some tables specify a translation start and end in addition to the transcription start and end. Some tables contain strand information, others don't. Most tables, but not all, specify a name for each element. Based on the format of the data described by a table, different query and output formatting options may be offered. Non-positional tables Non-positional tables contain data not tied to genomic location, for example a table that correlates a Known Gene ID with a RefSeq accession ID. Some non-positional tables relate internal numeric mRNA IDs to extended information such as author, tissue, or keyword. Some "meta" tables in this category contain information about the structure of the database itself or describe external files containing sequence data. Getting started - simple queries In its most basic form, the Table Browser can be used to retrieve a specific subset of records from a track or positional table in a selected genome assembly. The query may be based on a specific position or a set of one or more identifiers. This section describes the steps required to conduct basic simple data queries using the Table Browser. Once you have mastered the basic Table Browser functionality, refer to subsequent sections for information about generating more complex queries that use filters, intersections, and alternative data output formats. Simple position-based query Follow these steps to display a list of records that lie within a specific position in a table: Step 1. Pick a genome assembly Specify the genome assembly from which you'd like to retrieve the data by choosing the appropriate organism in the genome list, then selecting the assembly version from the assembly list. Note that the assembly list refreshes each time a different option is selected in the genome list. Assemblies are typically named after the first three characters of an organism's genus and species names. Step 2. Pick an annotation track The group list shows all the annotation track groups available in the selected genome assembly. The names correspond to the groupings displayed at the bottom of the Genome Browser annotation tracks page. When a group is selected from the list, the track list automatically updates to show all the annotation tracks available within that group. - If you already know the name of the annotation track in which you're interested, select the All Tracks option in the group list, then select the track from the track list. Similarly, you can directly select a table by choosing the All Tables option in the group list, selecting a database from the database list, then selecting the table from the table list. - To examine all the tracks available within a certain group (e.g., all gene prediction tracks), select the group name from the group list, then browse the entries in the track list. - Custom annotation tracks created during the current session are listed under the Custom Tracks group. - If no selections are made from the group or track lists, the track selection defaults to the Known Genes track in the Genes and Gene Prediction Tracks group. Step 3. Pick a table The table list shows all tables (both positional and non-positional) associated with the currently-selected track. By default, it displays the primary table for the track, i.e. the table containing the data shown in the Genome Browser annotation track. Other tables in the list are linked to the primary table by a common field and may provide supporting data used in constructing the annotation. - If the group list is set to the All Tables option, the tables list will show all tables present in the database currently selected in the database list, rather than those associated with a particular track. Step 4. Pick a genomic region (positional tables only) By default, the Table Browser region is set to genome, which will display all the data records in the selected table. - To restrict the data to a specific position range, type the position into the position box. Some examples of specific positions include a chromosome name (chrX), a coordinate range within a chromosome (chrX:100000-400000), or a scaffold name. - You can select multiple genomic regions by clicking the "define regions" button and entering up to 1,000 regions in a 3- or 4-field BED file format. - To look up the position range of a genomic element -- such as a gene name, an accession ID, an STS marker, etc. -- or keywords from the GenBank description of an mRNA, type the string into the position box, then click the Lookup button. - The data in non-positional tables are not tied to genomic coordinates; therefore, the region option is unavailable when a non-positional table is selected. A basic query on a non-positional table will show all the data in the table. Step 5. Display the output Click the Get Output button to display the results of the query. By default, the Table Browser outputs the data from all fields in the selected table as tab-separated text on the screen. See the Output formats section for information on configuring the query output. Example: Here is an example of a simple query that retrieves all the RefSeq Genes records in the position range chr7:26906938-26940301 on the May 2004 human genome assembly. 1. Select the Human option in the genome list. 2. Select the May 2004 option in the assembly list. 3. Select the Genes and Gene Prediction Tracks option in the group list. 4. Select the RefSeq Genes option in the track list. 5. Type chr7:26906938-26940301 in the position box (the Table Browser will automatically select the position option button). 6. Click the Get Output button. The Table Browser will display the records for the RefSeq accessions NM_005522, NM_153620, NM_006735, NM_153632, NM_030661, and NM_153631. Batch query using identifiers In many cases, you may want to retrieve data based on a list of one or more accessions, IDs, or names, rather than querying by genomic position. Many tracks in the Table Browser, such as those in the Genes and Gene Prediction or Variationtrack groups, support identifier queries. The identifier type used in the query must match the kind of identifiers present in the track data, e.g., mRNA accession IDs must be used to query the mRNA table and rsIDs must match those in the dbSNP table. Follow these steps to display a list of records that correspond to a set of accessions or names entered as query input. Step 1. Pick the genome assembly, track, and table Step 2. Select the genome region setting Step 3. Load the identifiers into the browser Click the Paste List button to type or paste in the identifiers or the Upload List button to load the data from a file existing on your local computer. - If you are loading multiple identifiers, entries must be separated by a space, tab, or line. - Wildcards may not be used in the list (see the Filter section for information about conducting queries that include wildcards). - The Table Browser will retain the identifier list until you delete the information by clicking the Clear List button. Step 4. Click the Get Output button See the Output formats section for information about configuring the query output. Batch query from positions If you have a list of genomic positions and want to retrieve information about their properties, you can use the Define Regions button to input multiple positions to query a chosen table. Please note, any items in the table that overlap with the defined regions will be included in the Table Browser output. In this example, you want to determine the dbSNP rsID names for your list of positions. Step 1. Select genome assembly and track To determine dbSNP rsIDs we will be using Human genome hg38 and dbSNP153. Step 2. Select the define regions button, enter regions You can find the define regions button under the Define region of interest section. Upload, type, or paste in your regions of interest, making sure they are in the desired 0/1 base notation. They will only be accepted in BED or positional format. Step 3. Select output format and get output If you want all data from a table, you need not change the output format from the default. If you want only particular columns from the table, you can change it to selected fields from primary and related tables. Once you hit the get output button, you will be redirected to a column selection page or if you did not change the output format, your output data itself. Get gene symbols in a query Follow the example below to obtain gene symbols in your query: - 1. Select the clade, genome, assembly, group, table, and region as desired. - 2. Change the output format to selected fields from primary and related tables. - 3. Click get output to go to the next step of selecting fields from related tables. - 4. Select the fields you would like from your primary table. - 5. On the same Select Fields form, find the table for the related kgXref table. For example, look for the hg38.kgXref table, and then check the checkbox next to Gene Symbol to add gene symbols to your query results. - 6. Click get output again to get the final query output. Filtering output by constraining field values The Table Browser filter option can be used to: - apply constraints on table field values to restrict which records should appear in the query output - conduct batch queries using wildcards - include fields from multiple tables in the query output Filtering on fields from a single table Follow these steps to create a filter on one or more fields in a single table: Step 1. Select the assembly, track, and region Step 2. Click the Create button on the filter line Step 3. Add the filter constraints One or more of the fields in the currently selected table may be filtered by typing constraints into the corresponding text boxes. - By default, the initial values set up in the filter match all records in the table. - Constraints must match the data type of the field to be applied successfully. For example, the geneName field in the hg17 refFlat table is a string; therefore, constraining values must also be strings. See the Filter constraints sections for more information on valid filter values. - Multiple filter values may be applied against one field by separating the values with spaces. - Individual field constraints are combined with AND, i.e. a record must meet the constraints on all fields to be retrieved. Step 4. Click the Submit button to apply the filter Once a filter has been created on a table, it will persist for the duration of the Table Browser session or until it has been cleared. Only one filter can exist for a table at a time, but multiple filters may exist in one session if they are applied on different tables. To modify an existing filter, click the Edit button on the filter line. To remove a filter, click the Clear button. Filtering on fields from multiple tables A Table Browser filter may include constraints on fields from tables related to the primary table. To create a filter composed of fields from multiple tables: Step 1. Select the assembly, track, and region Step 2. Click the Create button on the filter line Note: If a filter already exists on the table, click the Edit button to modify it or the Clear button to remove it. Step 3. Select the tables to include in the filter Scroll down to the Linked Tables section of the page. The tables listed in this section are linked to the selected table by one or more common fields (typically a name, accession, or ID field). Click the boxes in front of the table(s) whose fields you wish to include in the filter, then click the Allow Filtering Using Field in Checked Tables button. The fields of the selected tables will be displayed in the top portion of the page. Step 4. Add the filter constraints Step 5. Click the Submit button to apply the filter Note: In the current implementation of the Table Browser, the selected fields from primary and related tables output format option must be used when including fields from multiple tables in a filter. Check the boxes for all tables in the Linked Tables list on which filter constraints have been applied, then click the Allow Selection From Checked Tables button to include them in the output. Filter constraints Strings Text fields are compared to words or patterns containing wildcard characters. Valid wildcards are i "*" (matches 0 or more characters) and "?" (matches a single character). Each space-separated word or pattern in a text field box is matched against the value of that field in each record. If any word or pattern matches the value, then the record meets the constraint on that field. Numbers Numeric fields are compared to table data using an operator such as <, >, != (not equals) followed by a number. To specify a range, enter two numbers (start and end) separated by white space and/or a comma. Free-form queries When the filters on individual fields aren't sufficiently flexible, the free-form query text box allows the application of more complex constraints that typically relate two or more field names of the selected table. Valid free-form queries use the syntax of the SQL where clause (using wildcards as defined above). Free-form queries combine simple constraints with AND, OR, and NOT using parentheses as needed for clarity. A simple constraint consists of a table field name, a comparison operator (see below), and a value: a number, string, wildcard value (see below), or another field name. In place of a field name, you may use an arithmetic expression of numeric field names. - String or wildcard values for text comparisons must be quoted. Single or double quotes may be used. If comparing to a literal string value, use the "=" or "!=" operator. If comparing to a wildcard value, use the "LIKE" or "NOT LIKE" operator. - Numeric comparison operators include <, <=, =, != (not equals), >=, and >. - Arithmetic operators include +, -, *, and /. - Other SQL comparison keywords may also be used. Example: The following examples show free-form queries applied to the human refGene table). - txStart = cdsStart - searches for gene models missing expected 5' UTR upstream sequence (if strand is "+"; 3' UTR downstream if strand is "-") - chrom NOT LIKE "chr??" - restricts search to chromosomes 1 - 9, X and Y - cdsEnd - cdsStart) > 10000 - selects genes with coding spanning more than 10 kbp - txStart != cdsStart) AND (txEnd != cdsEnd) AND exonCount = 1 - finds single exon genes with both 3' and 5' flanking UTR - cdsEnd - cdsStart) > 30000) AND (exonCount=2 OR exonCount=3) - finds genes with long spans but only 2 - 3 exons Intersecting data from multiple tables It is often interesting to compare the positions of features in different annotation tracks to identify points of overlap. The Table Browser intersection utility can be used to generate various position-based comparisons of track features. Using the intersection utility, you can: - examine all genomic positions where the feature data from the two tracks overlap - identify genomic locations where there is no overlap between track features - establish thresholds for the amount of overlap that must exist between the two feature sets - conduct feature-by-feature comparisons as well as base-by-base comparisons of tracks - complement (invert) a position set before comparing the tracks An intersection may be expanded to include additional tables by using the Table Browser custom track feature. Note: The intersection utility can be used only on positional tables. To generate intersections incorporating data in non-positional tables, use the Table Browser filter utility. See the Filtering on fields from multiple tables section for more information. Intersecting data from two tables Follow these steps to configure and generate an intersection between two positional tables: Step 1. Select the assembly, track, table, and region for the primary table Note: Only positional tables may be used in an intersection. Step 2. Click the Create button on the intersection line Note: If an intersection already exists on the table, click the Edit button to modify it or the Clear button to remove it. Step 3. Select the secondary track to include in the filter Select a group in the group list, then select a track from the track list. To view all the tracks available, regardless of group, select the All Tracks option in the group list. Step 4. Select a combination method The Table Browser provides two major types of comparisons: - feature-by-feature comparisons preserve the structure of the primary table. For example, if the primary table describes exon structure and the features are compared with a second table, the results will describe exon structure (unless you choose an output format in which the structure is lost). - base-by-base comparisons examine the primary table and the table underlying the secondary track one base at a time. The structure of the primary table is not preserved in this comparison. For example, even if the primary table describes exon structure, the intersection results will contain only position ranges; no information about exon/block structure, strand, or translation region will be retained. Click the circle in front of a combination method to select it. Only one method may be selected from the two sets of methods. For more information about the individual combination options, see the Intersection Options section. Step 5. (optional) Select the complement options Check the box in front of one or both tables to complement the feature data. The complement options allow you to invert the set of positions covered by one or both tables. For example, if you choose to complement the primary track, any position covered by the that track's features will be considered not covered, and vice versa. This option provides more flexibility in comparing track positions. Step 6. Click the Submit button to apply the intersection Once an intersection has been created on a table, it will persist for the duration of the Table Browser session or until it has been cleared. Only one intersection may exist at a time. To modify an existing intersection, click the Edit button on the intersection line. To remove an intersection, click the Clear button. Intersecting data from more than two tables The Table Browser intersection utility limits combinations to only two tables. An existing intersection may be expanded to include additional tables by using the Table Browser custom track utility. To create an intersection on multiple tables: Step 1. Set up an intersection between two tables See the Intersecting data from two tables section for more information. Step 2. Save the intersection data in a custom track See the Saving data as a custom track section for information on generating a custom track. Note: In the current implementation of the Table Browser, you must use the Get Custom Track button on the custom track page to add the custom track to the Table Browser track list. Step 3. Select the newly-generated custom track Select the Custom Tracks option in the group list, then select the newly created custom track from the track list. Step 4. Create an intersection with another track Follow the steps in the Intersecting data from two tables section to intersect the custom track with another track. Intersection options Feature-by-feature comparisons Some comparisons preserve the primary table's gene and alignment structure, if it exists. For example, if the refGene table (human RefSeq Genes track) is combined with another table using one of these comparisons, the resulting output data will describe exon structure (unless you choose an output format in which the structure is lost). Primary table features are kept or discarded based on the amount of positional overlap with the features in the table underlying the secondary track. The Table Browser offers the following options in this category: - Any overlap: A primary table record will appear in the output if any of its base positions are covered by any feature in the secondary table. - No overlap: A primary table record will appear in the output only if none of its base positions are covered by any feature in the secondary table. - Overlap greater than a specified threshold: A primary table record will appear in the output if the percentage of its base positions covered by secondary table features is greater than the user-specified threshold. - Overlap less a specified threshold: A primary table record will appear in the output if the percentage of its base positions covered by secondary table features is less than the user-specified threshold. Note: If the primary table has an exon/block structure, only those bases located in exons and/or blocks will be counted. Base-by-base comparisons In these combination options, the positions of the primary and secondary table features are compared one base position at a time. When applying base-by-base comparisons, the structure of the primary table is not preserved. For example, if the refGene table (from the human RefSeq Genes track) is compared with a secondary table using these comparisons, the resulting output data will not describe exon structure. Instead, only position ranges will be returned; the exon/block structure, strand, and translation region information will be discarded. The Table Browser provides the following base-by-base combination options: - Base-by-base intersection (AND): A nucleotide position is included in the output if it is covered by at least one feature of both the primary table and the secondary table. - Base-by-base union (OR): A nucleotide position is included in the output if it is covered by at least one feature of either the primary table or the secondary table. Note: If the primary table has an exon/block structure, only base positions located in exons and/or blocks will be counted. Base-by-base complement (NOT) Before the Table Browser applies a feature-by-feature or base-by-base comparison to the table data, the set of positions covered by one or both tables can be inverted (complemented). When the data set of a table is complemented, any position covered by the table's features in the original data will be considered not covered in the inverted data, and vice versa. This option gives the user more flexibility in comparing table positions. Correlating data from two tables The Table Browser Correlation function creates a scatter plot of the data points of two tables as well as provides individual histograms of the data points from both tables. Additionally, it will also show a plot of the Residuals vs. Fitted which can be used to detect non-linearity, unequal error variances and outliers. The correlation function uses Pearson's correlation, which is optimized to work with continuous data such as wiggle tracks. For tracks that do not have data values such as gene-structured tracks, the data value used in the calculation is 1.0 for bases covered by exons and 0.0 at all other positions in the region. Due to memory and processing limitations, the number of data points that can be plotted is limited to 300,000,000. The "Window data to" function allows you to smooth out your plot by taking the average of the number of data points specified (defaults to 1). The total number of bases analyzed is independent of the data window. There is currently no way to output the results of the Correlation function. Output formats The data resulting from a Table Browser query may be configured in a number of different ways: - The output can be displayed on the screen, saved to a file, or saved to an annotation track table that can be displayed in the Genome Browser or used in a subsequent Table Browser query. - The data can include all fields from the primary or selected table, or can be restricted to selected fields from the primary table and related tables. - The data can be organized in one of several formats: tab-separated, sequence (FASTA), Browser Extensible Data format (BED), Gene Transfer Format (GTF), or a statistical summary of the data in the query. The output options available for a specific query may vary depending on the table(s) selected. For example, non-positional table data cannot be organized in a position-based format, but instead may be displayed only in tab-separated format. The Table Browser will automatically update the options on the output format list to show only those available for the current query. Displaying all fields in a table To display all the fields of the records in the query output in tab-separated format, select the all fields from primary table option. Displaying selected fields from one or more tables To restrict the query output to a subset of the fields in a table, choose the selected fields from primary and related tables option. You will be prompted to pick the table fields to display. Click the box in front of the fields you would like to see in the query output (or click the Check All button to select all the fields), then click the Get Fields button. To include data fields from other tables linked to the selected table, choose the selected fields from primary and related tables option, then scroll down to the Linked Tables section of the page. The tables listed in this section are linked to the selected table by one or more common fields (typically a name, accession, or ID field). Click the boxes in front of the table(s) whose fields you wish to include in the query output, then click the Allow Selection From Checked Tables. The fields of the selected tables will be displayed in the top portion of the page. Click the boxes in front of the fields that you wish to include in the query output, then click the Get Fields button underneath any of the field lists to generate tab-separated output that includes data from all the selected fields. Note that the Get Fields and Cancel buttons apply globally to all the selected tables, but the Check All and Clear All buttons apply only to the fields listed directly above the buttons. Displaying sequence (FASTA) data (positional tables only) To display the genomic sequence underlying the query results, select the sequence option in the output format list. The Table Browser will present you with several options to configure the output display. When you have completed the configuration, click the Get Sequence button. When displaying sequence data for gene prediction tracks, you will also be offered the option to view the protein and mRNA sequence as extracted from the data source in addition to the genomic sequence. Displaying CDS FASTA alignments (genePred tables only) The CDS FASTA alignments are created from a Multiple Alignment File (MAF) in combination with a genePred table. The UCSC MAF format stores multiple alignments at the DNA level between entire genomes. You can use the Table Browser to return FASTA alignments of coding regions in nucleotide-space or translated into amino acid-space. However, it is worth noting that the initial MAF files are all created by aligning genomes at the DNA level. Genome-wide CDS FASTA alignments Note that when using the Table Browser to fetch CDS FASTA output, it is best to restrict your query to a reasonable-sized position range rather than requesting output from the entire genome. A genome-wide query will take a substantial amount of compute time, and it is likely that your Internet browser will time out and disconnect. If you would like to download genome-wide CDS FASTA output for any of several model organisms, you can do so from the download server. Creating CDS FASTA alignments using the Table Browser To display FASTA multiple alignments for the CDS regions of genes, select the CDS FASTA alignment from multiple alignment option in the output format list. In order to see this output format option, you must have a genePred table selected. If you limit your search to a certain position range within the genome (rather than searching the entire genome), the tool will return FASTA alignments for all genes that overlap the position for which you are searching. The Table Browser will present you with a configuration page. On this page, you can select options for your output. First, select your MAF table. This is the table from which the multiple alignments will be extracted for the CDS regions of your gene track. If you do not know the name of the MAF table that corresponds to the Conservation track, you can find it in the Genome Browser by following these instructions. Then select any of the following choices: - Separate into exons - The default behavior is for the coding exons of each gene to be concatenated into one sequence in the output FASTA multiple alignment. In this case each output row header has the format listed below under "Whole gene format". If the separate into exons option is chosen then each exon will be listed with a separate header in the format listed below under "Exon format". - Show nucleotides - The default behavior is for the nucleotides in the alignment to be translated into amino acids according to the strand and exon frames defined in the selected genePred table. If this option is chosen, then the nucleotides in the alignment will not be translated into amino acids. - Output lines with just dashes - The default behavior is for the alignment rows that contain only dashes to not be printed. If this option is chosen, then these dashes-only rows are printed. - Format output as table - If this option is chosen, the header and sequence for each organism will appear on the same line. - Truncate headers as __ characters (enter zero for no headers) - This option works in conjunction with the "Format output as table" option. If you want to see only a portion of the headers, choose this option, and enter the number of characters at which you would like the headers truncated. Finally, from the list of species, select those that you would like included in the FASTA multiple alignment output. Press the "get output" button to view the output. Explanation of CDS FASTA header format Whole gene format: geneName_assemblyName peptideLength location Exon format: geneName_assemblyName_exonNum_totalExons exonLength inFrame outFrame location Here are the descriptions for each field name: - geneName- the name field from the genePred table. - assemblyName- the UCSC assembly name for the species. - peptideLength- the length of the entire coding region. If the "Show nucleotides" option is chosen, this will be in nucleotides, otherwise it will be the number of amino acids in the peptide. - location- this is the chromosome position within the assembly that is aligned in the multiple alignment. The format of this string is chrom:start-end followed by the strand where the alignment occurs. If more than one region is aligned then all the regions are listed with a semi-colon (;) between each position. This address is in genome browser coordinates (i.e. the start address is one-based). - exonNum- the ordinal of the exon. Exons are counted starting at one and begin at the transcription start site and progress along the strand of transcription. - totalExons- the number of coding exons in the gene. - exonLength- the length of the current exon. If the "Show nucleotides" option is chosen, this will be the number of nucleotides in the exon, otherwise it will be the number of amino acids in the exon (with amino acids translated from split codons placed in the exon where two of the three nucleotides lie). - inFrame- the frame number of the first nucleotide in the exon. Frame numbers can be 0, 1, or 2 depending on what position that nucleotide takes in the codon which contains it. - outFrame- the frame number of the nucleotide after the last nucleotide in this exon. Frame numbers can be 0, 1, or 2 depending on what position that nucleotide takes in the codon which contains it. Explanation of CDS FASTA sequence format As noted above, the CDS FASTA output files can be in either DNA-space or protein-space. In some instances, there is a dash ("–") in the sequence portion of the CDS FASTA file. Dashes are used in several circumstances. They indicate missing sequence for the aligning genome, as well as deletions in the aligning genome or insertions in the base genome. Because the CDS FASTA alignments are based on one reference genome, any amino acids or nucleotides that are not in the reference genome are not displayed. Consequently the peptides shown for aligning genomes are not necessarily the peptide that the gene of the other organism would generate. Any sequence inserted in an aligning genome or deleted in the base genome will not be present in the alignment. We represent this condition with an orange bar in the Genome Browser display, but the CDS FASTA alignments silently ignore this issue. Nucleotide CDS FASTA sequence: Consider the example below that shows the FASTA sequence for four species aligned with the first exon of the human gene PLEKHO1 (UCSC Gene: uc001ett.1). Note that the rat (rn4) row is missing the first three nucleotides. This could be due to a lineage-specific insertion between the rat and human genomes, or a lineage-specific deletion between the human and rat genomes. Note also that the Zebrafish (danRer4) row contains only dashes. This could be due to excessive evolutionary distance between the zebrafish and human, missing data in the zebrafish, or independent indels in the region in both species. Sometimes it is helpful to view the Conservation track in the Genome Browser in this area to clarify the exact meaning of the dashes. >uc001ett.1_hg18_1_6 30 0 0 chr1:148389072-148389101+ ATGATGAAGAAGAACAAcode >uc001ett.1_panTro2_1_6 30 0 0 chr1:129156502-129156531+ ATGATGAAGAAGAACAAcode >uc001ett.1_rn4_1_6 30 0 0 chr2:190795892-190795918- ---ATGAAGAAGAGCGGCTCCGGCAAGCGG >uc001ett.1_danRer4_1_6 30 0 0 ------------------------------ >uc001ett.1_oryLat2_1_6 30 0 0 chr11:3404940-3404969- AGGATGAAGAAAAGCAACCAGAGCAGGCGG Amino Acid CDS FASTA sequence: - Codons that have a dash in any of the three nucleotides are represented by a dash in the amino acid. - Codons with an N in any position are represented with an X. - Stop codons are represented with a Z. - All other amino acids follow the IUPAC amino acid codes. - In exon format, when the codon triplet is split between two exons, the amino acid will be displayed as part of the exon containing two of the three nucleotides like so: Saving query results in GTF or BED format (positional tables only) To format the query results using GTF or BED conventions, select the corresponding option in the output format list. Note that when you select GTF, the table browser translates the output into this format. For tables that lack feature designations, all records are arbitrarily assigned the feature "exon" to conform to GTF specifications. If you select BED format, you will be presented with the option to include and configure a custom track header and options for organizing the data. When you have finished the configuration -- or to accept the default options -- click the Get BED button at the bottom of the window. To understand the name column in the BED format, see this FAQ. Saving data to a file By default, the Table Browser displays query results directly in your internet browser window. To redirect the data to a file, type a file name into the output file box before starting the query. The Table Browser will prompt you for the location of this file on your local disk while processing the query. Saving data as a custom track (positional tables only) Query output may be saved in a format that can be displayed as a custom annotation track in the Genome Browser. Custom tracks created during a Table Browser session may also be used for subsequent queries and intersections in the same session. For more information on custom tracks, see the Genome Browser User's Guide. To save query data in custom track format, select the custom track option in the output format list. When the query is executed, the Table Browser will prompt you to customize the track header and configure the record layout of the data. The configuration is optional; the Table Browser automatically sets up a default track configuration. Click the Custom track link for more information on custom track syntax and format. When you have finished configuring the custom track -- or to accept the default configuration -- click one of the buttons at the bottom of the window to create the custom annotation track. - To display the query results as text on the screen, click the Get Custom Track File button. - To save the query results to a file on your local disk for future use, specify a file name in the output file box before executing the query, then click the Get Custom Track File button. - To load the query results into a table accessible from the Table Browser table list, click the Get Custom Track in Table Browser button. - To view the query results as a custom track in the Genome Browser, click the >Get Custom Track in Genome Browser button. Your browser display will be redirected automatically to the Genome Browser, with your custom track positioned near the top of the annotation tracks window. - To access your custom track data in a subsequent query in the same Table Browser session, select the Custom Tracks option from the group list to display the custom tracks available. Displaying query results as Genome Browser hyperlinks (positional tables only) To examine the records in the query output individually in the Genome Browser, select the hyperlinks to Genome Browser output option. The Table Browser will display a list of one or more hyperlinks corresponding to the individual records in the output data. Click a link to open up the Genome Browser display to the item and position shown on the hyperlink. Displaying a statistical summary of query data (positional tables only) To generate a statistical summary of the query output data, the region covered by the query, and the CPU time required to process the query, click the Summary/Statistics button. Video examples of Table Browser queries Finding a list of genes in a genomic region []       Visit our Video Page.       Visit our YouTube channel. Obtaining coordinate sequences for a gene exon []       Visit our Video Page.       Visit our YouTube channel. Finding all the SNPs in a gene []       Visit our Video Page.       Visit our YouTube channel. Finding SNPs upstream of a gene []       Visit our Video Page.       Visit our YouTube channel. 
/goldenPath/help/hubQuickStartSearch.html:Track_Hub_Quick_Start	Searchable Track Hub Quick Start Guide Track Hubs are a method of displaying remotely-hosted annotation data quickly and flexibly on any UCSC assembly or remotely-hosted sequence. Making your annotation data searchable is an important improvement to the usability of your hub, especially if your annotations are not otherwise represented on the Browser. This Quick Start Guide will go through making a searchable track hub from a GFF3 file; converting to a genePred, bed, and bigBed, then creating a trix search index file. This example will be made with the new "useOneFile" feature to avoid any need for separate genome.txt and trackDb.txt files. STEP 1: Downloads Gather our settings and data files in a publicly-accessible directory (such as a university web-server, CyVerse, or Github). For more information on this, please see the hosting guide. Copy the hub.txt file using wget, curl, or copy-paste: wget http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubSearchable/hub.txt Download some example GFF3 data from Gencode. This file happens to be long non-coding RNAs (lncRNAs): wget ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_32/gencode.v32.long_noncoding_RNAs.gff3.gz Next, you will need to download four Genome Browser utilities to convert the GFF3 file to bigBed format and run the search index command. Similar commands exist to convert other file types. These are operating system specific: Utility Name MacOS Download Linux Download ---------------- ---------------- ---------------- gff3ToGenePred Download Download genePredToBed Download Download bedToBigBed Download Download IxIxx Download Download STEP 2: Format Data In order to format the data, you will need to run a command to make those commands executable: chmod +x gff3ToGenePred genePredToBed bedToBigBed IxIxx Then run the first conversion from GFF3 to genePred, making sure to include -geneNameAttr=gene_name so that gene symbol is used as the name2 instead of ID number, and sorting by chromosome and position: gff3ToGenePred -geneNameAttr=gene_name gencode.v32.long_noncoding_RNAs.gff3.gz stdout | sort -k2,2 -k4n,4n > gencode.v32.lncRNAs.genePred Convert that genePred file to a bed file: genePredToBed gencode.v32.lncRNAs.genePred gencode.v32.lncRNAs.bed Compress and index that bed file into a bigBed format, adding the -extraIndex=name to allow EnstID searches: bedToBigBed -extraIndex=name gencode.v32.lncRNAs.bed https://genome.ucsc.edu/goldenPath/help/hg38.chrom.sizes gencode.v32.lncRNAs.bb If you would like to stop here, you will be able to display your bigBed hub and search for the names that were indexed into the bigBed file (EnstID). You will not be able to use the searchIndex and searchTrix trackDb setting, which require creating a key and value search index for your file as shown below. STEP 3: Create Search Index If you want to link your annotation names to anything other than the field referrenced in the -extraIndex command, you will need to make and index file. We will make an input file which will link one identifier (EnstID) with search terms composed of gene symbols and EnstIDs. Below is one example of a command to create an input file for the search indexing command: cat gencode.v32.lncRNAs.genePred | awk '{print $1, $12, $1}' > input.txt To examine or download that file, you can click here. Note that the first word is the key referenced in the BED file and the following search terms are associated aliases will be searchable to the location of the key. These search terms are case insensitive and allow partial word searches. Finally you will make the index file (.ix) and the index of that index (.ixx) which helps the search run quickly even in large files. ixIxx input.txt out.ix out.ixx STEP 4: View and Search Enter the URL to your hub on the Connected Hubs tab of the Track Data Hubs page. Alternately, you can enter your hub.txt URL in the following web address: genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&hubUrl=YourUrlHere If you would like to look at an already-made example, click the following link which includes hideTracks=1 to hide other tracks. After the link is a picture of what the hub should look like: https://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&hideTracks=1&hubUrl=http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubSearchable/hub.txt [A display of the Searchable hub track] Once your hub displays, you should be able to type in a gene symbol or Enst ID and scroll down the results page until you see your search results. [Typing a search term in the search box] You can type your search term (fam87b) in the box above the ideogram and press go. Note that it is not case sensitive. Scrolling to the bottom of the search results page, you will see your searchable hub keyword that was linked with your search term. Clicking into it will bring you to the position of your search term. [Search hit for fam87b] [Search results for fam87b] If you are having problems, be sure all your files are publicly-accessible and that your server accepts byte-ranges. You can check using the following command to verify "Accept-Ranges: bytes" displays: curl -IL http://yourURL/hub.txt Note that the Browser waits 5 minutes before checking for any changes to these files. When editing hub.txt, genomes.txt,and trackDb.txt, you can shorten this delay by adding udcTimeout=1 to your URL. For more information, see the Debugging and Updating Track Hubs section of the Track Hub User Guide. Understanding hub.txt with useOneFile The hub.txt file is a configuration file with names, descriptions, and paths to other files. The example below uses the setting useOneFile on to indicate that all the settings and paths appear in only the hub.txt file as opposed to having two additional settings files (genomes.txt and trackDb.txt). Please visit the UseOneFile guide for more information. The most important settings to make the hub searchable appear in the third section, in what would formerly be the trackDb.txt file. The searchIndex and searchTrix indicate which fields are indexed in the bigBed file and where to find the .ix file respectively. To see the actual hub.txt file for the above example, click here. hub MyHubsNameWithoutSpaces shortLabel My Hub's Name longLabel Name up to 80 characters versus shortLabel limited to 17 characters email myEmail@address descriptionUrl aboutMyHub.html useOneFile on genome assembly_database_2 track uniqueNameNoSpacesOrDots type track_type bigDataUrl track_data_url shortLabel label 17 chars longLabel long label up to 80 chars visibiltiy hide/dense/squish/pack/full searchIndex field,field2 searchTrix path/to/.ix/file Additional Resources - Track Hub User Guide - Guide To useOneFile setting - Search file .ix documentation - Mailing list question with searchable Track Hub - Mailing list question with searchable Custom Tracks - Track Database (trackDb) searchTrix Definition - Quick Start Guide to Organizing Track Hubs into Groupings - Quick Start Guide to Assembly Track Hubs 
/goldenPath/help/mirror.html:Genome_Browser_Manual_Installation	Installation of a UCSC Genome Browser on a local machine ("mirror") Contents Considerations before installing a Genome Browser Installing a Genome Browser locally with the GBiC installer Docker installation instructions Manual installation instructions Using UDR to speed up downloads The genome-mirror mailing list Considerations before installing a Genome Browser Like most web servers, running a Genome Browser installation at your institution, even for your own department, requires a Unix machine, disk space (6TB for hg19), and the resources to update the site and underlying OS regularly. You may want to consider these alternatives before embarking on a full UCSC Genome Browser installation directly on your server. For information about operating in the cloud, visit the Cloud Data and Software Resources help page. 1. Embed the Genome Browser graphic in your web page If you only want to include a genome browser view into your webpage for already existing genomes, you can use an <iframe> tag and point it to http://genome.ucsc.edu/cgi-bin/hgRenderTracks, which will show only the main browser graphic without any decorations. You can then use various parameters to adapt this graphic to your use case (e.g. set the displayed position, switch tracks on/off or highlight a range with a color), see our manual pages for a list of the parameters. 2. Use Assembly Hubs Assembly Hubs: Assembly Hubs allow you to prepare any FASTA file, add annotations and use the Genome Browser to visualize it. All you need is a webserver where you save the indexed genome sequence and files to annotate it, e.g. in BAM, bigBed or bigWig format. - Pros: - No installation, no updates, no long-term UNIX support necessary. - Stable for many years, the link to the assembly hub can be put into a manuscript. - For commercial users, no license is required. - Cons: - You need access to a public webserver to store the files. - Data is rendered at UCSC. Private data can be password-protected and loaded through https and restricted to UCSC's IP address, but has to be located on the web and accessible by the UCSC webserver. - For BLAT searches in your genome, you have to start a BLAT server yourself on a Unix server. - If your hub includes a high number of annotation files or HAL (multiple alignment) files and is located far from Santa Cruz, the display performance of assembly hubs may be slower than a local mirror. This issue can be resolved by using a UCSC mirror closer to the assembly hub (e.g. use genome-euro.ucsc.edu for assembly hubs located on servers in Europe, or genome-asia.ucsc.edu for those in Asia). Alternatively, you can improve performance by moving your hub data to a webspace provider closer to Santa Cruz or by using a content distribution network, where all content is mirrored and the closest location is chosen by your provider. 3. Use Genome Browser in a Box Genome Browser in a Box (GBiB): is a fully configured virtual machine that includes Apache and MariaDB, and has behavior identical to the UCSC website. GBiB loads genome data from the UCSC download servers on the fly. Website and data updates are applied automatically every two weeks. By default, GBiB uses the VirtualBox virtualization software, so it can be run on any operating system, Windows, OSX and Linux. It does not require VirtualBox, the virtual machine image can easily be converted to e.g., VMWare or HyperV. For increased privacy, you can download the genomes and annotation tracks you need and use your GBiB off-line even on a laptop. - Pros: - Relatively simple to install: download and double-click. - By default, software and data updates are managed remotely by UCSC, via an update script run every week. - Best performance when rendering local BAM/bigWig/bigBed files. - No Unix webserver needed, runs on any OS. - For commercial users: easier click-through licensing compared to a full multi-seat, manual license. - Cons: - Requires the free VirtualBox software or a commercial Virtualization system. - By default requires opening at least three outgoing ports to UCSC for MariaDB, Rsync downloads and BLAT in your firewall. Once all data is downloaded, no open ports are needed. - For maximum browsing speed, can require up to 2-6TB to store all genome annotation tracks, like a manual local installation. 4. Install locally with the Genome Browser installation script (GBIC) We recommend this only if none of the above options fulfill your needs. Our GBIC installation script will install a full local mirror of the UCSC website, for the assemblies you select. GBiC can also be used as a Docker container. See the Docker installation instructions below. We support mirror site installations as time allows, and have many functional mirrors of the Genome Browser worldwide. For details, see the section below. - Pros: - Relatively simple to install on a virtual machine or cloud instance: just run the script. - Best performance when rendering local BAM/bigWig/bigBed files. - For commercial users: easier click-through licensing compared to a full multi-seat, manual license. - Cons: - To keep up with changes in the Genome Browser, you will have to install linux packages and update the linux distribution yourself in the future and run the installation script with the 'update' command if you want to take advantage of new features in the Genome Browser. - Preferably run on a linux server that is not used otherwise. - By default requires opening at least three outgoing ports to UCSC for MariaDB and BLAT in your firewall. Once all data is downloaded and BLAT setup locally, no open ports are needed anymore. - For maximum browsing speed, can require up to 2-6TB to store all genome annotation tracks. 5. Install manually yourself, by following installation instructions We do provide step-by-step instructions for local installation, but do not recommend this, if any other system works for you. The internet also has various pages with instructions, but they are often out of date and may cause problems later. For details on manual instructions, see the section below. - Pros: - You will understand the complete setup of the Genome Browser. - For commercial users: license agreements can be customized to your needs. - Cons: - Not easy to setup, even for experienced Unix administrators. - Will probably require some support via the genome-mirror mailing list. - To keep up with changes in the Genome Browser, you will have to install linux packages and update the linux distribution yourself in the future and apply UCSC data updates yourself using rsync or MariaDB table loads - Configuration changes on our side may break your setup. - For maximum browsing speed, can require up to 2-6TB to store all genome annotation tracks. - For commercial users: license agreements take longer to negotiate, no click-through license. A license is required for commercial download and/or installation of the Genome Browser binaries and source code. No license is needed for academic, nonprofit, and personal use. To purchase a license, see our license Instructions or visit the Genome Browser store. Installing a Genome Browser locally with the GBiC installer If you do not want to use our prepared virtual machine Genome-Browser-in-a-Box, we provide a Genome Browser in the Cloud (GBiC) installation program that sets up a fully functional mirror on all major Linux distributions. The GBiC program has been tested and confirmed to work with Ubuntu 18/20/22/24 LTS, Rocky 9.5, and Fedora 30/35/41. Preferably, the installation should be performed on a fresh Linux installation, as it deactivates the default site config file in Apache and fills the MariaDB directory with numerous databases. The easiest way to accomplish this is to run the Genome Browser in the Cloud program in a new virtual machine. The program also works on Docker and cloud computing virtual machines, and has been tested on those sold by Amazon, Microsoft and Google. Like GBiB, the mirror installed by the GBiC can load the annotation data either from UCSC or a local database copy. If you load data from UCSC and use a cloud computing provider, it is highly advisable to run your instances in the US West Coast / San Francisco Bay Area or San Jose data centers; otherwise data-loading may be very slow. To run the installation program, please see the GBiC user guide. Docker installation instructions Download the Dockerfile to a new directory and build the docker image. This works on Windows, OSX and Linux, as long as Docker is installed: mkdir browserDocker && cd browserDocker wget https://raw.githubusercontent.com/ucscGenomeBrowser/kent/master/src/product/installer/docker/Dockerfile docker build . -t gbimage You can then run the gbimage image that you just built as a new container in daemon mode (-d) and export its port to localhost: docker run -d --name genomeBrowser -p 8080:80 gbimage The Apache server is running on port 8080 then and you should be able to access it via https://localhost:8080 More details on installing Docker can be found on the Docker help page. Manual installation instructions See mirrorManual.html: If the installation program does not work on your linux distribution or you prefer to make adaptations to your mirror, we provide these step-by-step installation instructions that cover the configuration of Apache, MariaDB, the Genome Browser CGIs, temporary file removal and other topics, like data loading through proxies. The following external websites were not created by UCSC staff and are of varying quality, but may be helpful when installing on unusual platforms. - Installation on Ubuntu - Installation into a FreeBSD jail, by Norbert Grundmann, Universitaet Muenster, Germany - Compilation of the Kent source tree on OSX with fink by Casey Bergman, Manchester, UK - Browser installation and new genome setup, by Oliver Elliott, Columbia University, USA - Installation notes in our wiki Using UDR to speed up downloads UDR (UDT Enabled Rsync) is a download protocol that is very efficent at sending large amounts of data over long distances. UDR utilizes rsync as the transport mechanism, but sends the data over the UDT protocol. UDR is not written or managed by UCSC. It is an open source tool created by the Laboratory for Advanced Computing at the University of Chicago. It has been tested under Linux, FreeBSD and Mac OSX, but may work under other UNIX variants. The source code can be obtained through GitHub. When using the GBIC installation program, the -u option will use UDR for all downloads. If you manually download data only occasionally, there is no need to change your method; continue to visit our download server to download the files you need. This new protocol has been put in place primarily to facilitate quick downloads of huge amounts of data over long distances. With typical TCP-based protocols like http, ftp, and rsync, the transfer speed slows as the distance between the download source and destination increases. Protocols like UDT/UDR allow for many UDP packets to be sent in batch, thus allowing for much higher transmission speeds over long distances. UDR will be especially useful for users who are downloading from places distant to California. The US East Coast and the international community will likely see much higher download speeds when using UDR vs. rsync, http or ftp. If you need help building the UDR binaries or have questions about how UDR functions, please read the documentation on the GitHub page and if necessary, contact the UDR authors via the GitHub page. We recommend reading the documentation on the UDR GitHub page to better understand how UDR works. UDR is written in C++. It is Open Source and is released under the Apache 2.0 License. In order for it to work, you must have rsync installed on your system. For your convenience, we offer a binary distribution of UDR for Red Hat Enterprise Linux 6.x (or variants such as CentOS 6 or Scientific Linux 6). You'll find both a 64-bit and 32-bit rpm here. Once you have a working UDR binary, either by building from source or by installing the rpm, you can download files from either of our our download servers in a fashion similar to rsync. For example, using rsync, all of the MariaDB tables for the hg19 database can be downloaded using either one of the following two commands: rsync -avP rsync://hgdownload.soe.ucsc.edu/mysql/hg19/ /my/local/hg19/ rsync -avP hgdownload.soe.ucsc.edu::mysql/hg19/ /my/local/hg19/ Using UDR is very similar. The UDR syntax for downloading the same data would be: udr rsync -avP hgdownload.soe.ucsc.edu::mysql/hg19/ /my/local/hg19/ The genome-mirror mailing list For questions about installing and mirroring the UCSC Genome Browser, contact the UCSC mailing list genome-mirror@soe.ucsc.edu. Messages sent to this address will be posted the moderated genome-mirror mailing list, which is archived on a SEARCHABLE PUBLIC Google Groups forum. 
/goldenPath/help/hgGeneGraph.html:Gene_and_Pathways_Interaction_Graph_User&#39;s_Guide	Gene and Pathways Interactions Graph User's Guide Contents Introduction Configuring the Pathway and Gene Interaction Display Data Sources and Methods Data Access Credits References [Gene Interaction Graph] Introduction The Pathways and Gene Interactions graph accompanies the "Gene Interactions" track and displays a detailed gene interaction and pathway graph based on data collected from two sources: curated pathway/protein-interaction databases and interactions found through text mining of PubMed abstracts. The curated data were imported from 23 pathway or protein-interaction databases (see the Methods section below). Curators at these databases typically read research articles, collect protein interactions from them and store them in a web-accessible database. Pathway databases such as Reactome or WikiPathways describe a whole set of interactions, e.g. the WNT pathway, and the type of effect, and sometimes annotate indirect or inferred effects as an interaction. They often work from review articles. In contrast, protein interaction databases focus more on the original literature that describes the results of the biochemical assay and focus less on the effect or direction of the interaction. The text mining data was generated in collaboration with the Microsoft Research Project Hanover Team using Literome machine-reading. Literome is a natural-language processing (NLP) system that analyzes sentences and tries to extract names of proteins and the type of interaction. A simple example is a sentence such as, "PTEN negatively regulates AKT3", which gets transformed to "PTEN-AKT3" and "regulation: negative". The text mining system was run on all 20 million PubMed abstracts at the end of 2014 and can also be queried through the website Literome. Configuring the Pathway and Gene Interaction Display Clicking on an item in the track display takes you to a page that includes a gene interaction graph with detailed information on the directionality and support for the various interactions displayed. The graph is initially centered on the gene clicked in the track display, with this gene highlighted in yellow. For example, you can see that the primary gene "SOD1" is highlighted in yellow in this image: [Gene Interaction Graph] By default, only the top 25 best-supported interactions are displayed, but this number can be increased, decreased or filtered using the controls above the image. The interaction display can be filtered using the drop-down menu to display subsets by their support: - All interactions, regardless of support type - Only interactions with some database support - Only interactions with pathway database support Genes in the interaction graph are connected by a number of different line types, with each type of line and the line properties themselves indicating different levels of support from text mining and databases. - Solid grey lines - only text-mining support for this interaction, with the thickness of the line indicating the number of articles supporting it. - Dashed blue lines - at least one curated database supports this interaction. - dark blue - the information is derived from a paper describing fewer than 10 interactions - light blue - the information is derived from a high-throughput paper, describing more than 10 interactions, e.g. a complex or a mass-spec study - Solid blue lines - both databases and text mining support this interaction Here you can see nearly all of the different types of lines in a single gene interaction graph centered around the ROBO3 gene: [Gene Interaction Graph Line Types] Lines may include arrows showing the directionality of this interaction. In these cases, the directionality is determined by majority support. For example, imagine an interaction between protein A and protein B; two articles support that A acts on B while a single article supports the opposite, B acting on A. In this case, because there are more articles supporting A acting on B, then the arrow will be drawn such that it starts at A and points to B. From the "Annotate Genes" drop-down, you can annotate genes based on GNF2 average expression, drugability from DrugBank entries, cancer type in the COSMIC Cancer Gene Census, and the number of non-silent mutations identified by the PanCancer analysis project. For the GNF2 expression and PanCancer Mutation coloring, genes will be colored on a sliding scale from light grey to black, with those items with the highest expression or the largest number of non-silent mutations being colored the darkest and those with lower expression or fewer mutations being colored grey. Genes will be colored dark blue if there is no information in the database. In this image, you can see a set of 14 genes that interact with TP53 colored by their PanCancer Mutation number: [Gene Interaction Graph 'Annotate Genes' Example] You can mouse-over items in the display to show more details about the gene such as their product. If you've chosen to annotate genes with one of the various databases, then it will display that information as well. For instance, hovering over the BAX gene in this example displays a description of the gene product as wells as the number of Pan-Cancer mutations since that option is selected: [Gene Interaction Graph Item Hover Example] You can mouse-over the connecting lines between genes to see more details about the evidence that supports this connection. In this image, you can see the details that pop-up when you mouse over such a line; information displayed includes database support and text-mining support. [Gene Interaction Graph Line Hover Example] If you click on the line connecting two proteins, you can see a SumBasic-selected snippet of text from a Pubmed abstract and, if it is a curated interaction, the supporting information from the pathway or interaction databases. This example shows the text-mined support for an interaction between CASP5 and HUNK: [Gene Interaction Graph Line Click Example] Below the graph of gene interactions and pathways, there is table of less-supported interactions. These are interactions which were mentioned only a few times each in the literature. [Gene Interaction Graph Extra Interactions] The numbers shown on mouse-over for each interaction represents the number of articles and number of databases that support this interaction. You can export the currently displayed gene interaction graph in a variety of formats including PDF, SVG, Cytoscape, and JSON. The gene interaction graph can be recentered around a new gene in a few different ways: (1) clicking a gene in the existing interaction graph, (2) clicking the triangle next to a gene in the table of minor interactions below the graph, (3) searching for a gene name in the search box above the graph. Data Sources and Methods Human protein interactions from the following databases were imported: Protein interactions - iRefIndex 13 which includes BIND, BioGRID, CORUM, DIP, HPRD, InnateDB, IntAct, MatrixDB, MINT, MPact, MPIDB and MPPI - Androgen Responsive Gene Database. This database is not available anymore on the internet, but we kept a copy. - String 9.1 - Negatome 2.0 - Corum Protein Complexes - Gene Ontology Protein Complexes Pathways - KEGG, Version from April 2011. This is a version of the database before the switch to a non-commercial license. - NCI Pathway Interaction Database. This database is not available anymore on the internet in its original format, but we kept a copy. - BioCarta. This database is not directly available in a machine readable format. We use a version from 2009 that was included in the original NCI-PID. As the original file is not available anymore, we provide a copy. - Reactome 2014 - WikiPathways, version 20170510 - OpenBEL large corpus, version 20150611 (commit 5515fcf, Jan 2016). This database is Copyright 2011-2015, Selventa and under a non-commercial license. - FastForward The quantitative contribution of each database in terms of number of gene-pairs is available here. For text mining, PubMed abstracts were downloaded from the National Library of Medicine (NLM) website. The abstracts were then tokenized and parsed syntactically using the SPLAT toolkit. Protein and Gene names were identified and normalized after which potential interactions were extracted using the Microsoft Research NLP "Protein and Pathway Extractors". The results were then mapped to the genome using their HGNC gene symbols. Text-mining results supporting by only a single abstract are in the database tables but are not shown in the user interface. Data Access The raw data for these graphs can be accessed in multiple ways. They can be explored interactively using the Table Browser, by selecting "group" - "All Tables" and "database" - "hgFixed". Under "table", select "hgFixed.ggLink". You can then start to explore the relationships between the database tables using the "data format description" button or download tables with "get output". All database tables related to this viewer start with the prefix "gg". The database tables can also be accessed programmatically through our public MariaDB server or downloaded from our downloads server for local processing. The database tables are: - ggLink - one row per gene/gene interaction. The field "minResCount" is the minimum number of interactions obtained from the same supporting article. E.g. if it is 10, then out of all supporting articles, there is one with 10 interactions curated from it and maybe others with more interactions. A cutoff of 50 should remove high-throughput data from the table. Note that while most databases are in the format source -> target, in this table, the target comes first and the source second. Gene names are separated by the "|"-symbol. - ggLinkEvent - connections between a ggLink and one of the ggEvent tables. The prefix of the eventId indicates the table: ppi/pwy links to ggEventDb, msr links to ggEventText. - ggEventDb - information about gene/gene interactions imported from protein interaction or pathway databases. The structure is modeled after the NCI PID interactions data schema and distinguishes genes, complexes and compounds on each side of the reaction, the type of the relation and contains the curated display names for the genes. The compounds are part of the table but not shown in our user interface. - ggEventText - information about gene/gene interactions obtained from text mining. - ggDocEvent - connections between documents and events. - ggDoc - information about documents referenced from ggEventText and ggDocEvent. - ggGeneClass - the HPRD/Panther class, one for each gene symbol. - ggGeneName - the HGNC name, one for each gene symbol. For more details about the tables and their fields, use the Table Browser's "describe schema" button. The annotations (GNF2 average expression, DrugBank, etc.) for genes are accessed as text files for performance reasons and can be downloaded from our downloads server. Credits - The text-mined data for the gene interactions and pathways were generated by Chris Quirk and Hoifung Poon as part of Microsoft Research Project Hanover. - Pathway data was provided by the databases listed under methods. - Thanks to Ian Donaldson for IRefIndex, the biggest and free collection of protein interaction databases. - Arjun Rao (UCSC) provided the ArgDB converter. - Thanks to Dexter Pratt for help with OpenBEL and to Charles Tepley Hoyt for the pybel converter. - Thanks to Alexander Pico for help with the WikiPathways data format GPML - The short gene descriptions are a merge of the HPRD and PantherDB gene/molecule classifications. Thanks to Arun Patil from HPRD for making them available as a download. - The track display and gene interaction graph were developed at the UCSC Genome Browser by Max Haeussler. References Poon H, Quirk C, DeZiel C, Heckerman D. Literome: PubMed-scale genomic knowledge base in the cloud Bioinformatics. 2014 Oct;30(19):2840-2. PMID: 24939151 
/goldenPath/help/hgIndelDisplay.html:Genome_Browser_Indel_Display	Alignment Insertion/Deletion Display Options The Genome Browser offers several ways to highlight gaps in alignments of query sequences (usually transcripts) to the genome. Gaps result from sequences in the genome or query (or both) that cannot be aligned. Legend - double horizontal line (=): both the genome and query have unalignable sequence between regions of aligned sequence, a double-sided insertion. If this option is not selected, it will display as a single horizontal line. - orange lines or purple lines: unalignable query sequence, orange for the middle of a sequence and purple for the beginning or end. - green lines: poly-A tail or poly-T head that does not align to the genome. Details When zoomed out past the base level, the browser chooses one color to represent many bases. The priority of display, from most important to least important, is: different mRNA base/nonsynonymous codon coloring (if enabled) or different item bases (if enabled), unalignable query sequence (orange or purple), an insertion in both genome and query (double horizontal line), and a poly-A tail (green). The browser will not display genomic/mRNA codon coloring when viewing large regions of the genome. Interpretation of Display Gaps are usually due to a deletion or insertion in one or both sequences, or, infrequently, a problem in the sequencing. Often, the genome will have a large "insertion" relative to a query (single horizontal line) that is actually an intron. Double-sided insertions (double horizontal line) are unusual and may indicate an assembly error, sequencing error, or polymorphism. Unalignable query sequence in the middle of a query (orange) implies extra bases in the transcript sequence or missing bases in the genome sequence. Insertions at the beginning or end of a query (purple), implies a partial alignment of the query. For instance, a very short sequence next to large intron gap may be incorrectly aligned. Unalignable query sequence may also be due to polymorphisms. Poly-T heads result from queries that are the reverse complement of the genomic sequence. Poly-A tails and poly-T heads (green) of mRNAs usually can not be aligned to the genome; this is a special case of an unalignable query sequence. For information about mRNA codon and base coloring, click here. For information about EST base coloring, click here. 
/goldenPath/help/hg18ToHg19LiftOver.html:Genome_Browser_LiftOver	LiftOver of tracks from hg18 to hg19 The tracks indicated by the logo [hg18 LiftOver logo] or [hg19 LiftOver logo] have been lifted from hg18 or hg17, respectively, with a minimum of quality control scrutiny. They are provided to our users with the intent that they assist in interpretation of other data, but must be used with caution. Not all annotations remain intact when lifted in this manner and in any case, cannot by definition contain any sequence that is new to hg19. It should also be noted that tracks containing large regions will lift less well because of the increased chance of spanning a region that changed between the two assemblies. 
/goldenPath/help/wiggle.html:Genome_Browser_Wiggle_Track_Format	Wiggle Track ASCII Text Format (.wig) Wiggle files and its bedgraph variant allow you to plot quantitative data as either shades of color (dense mode) or bars of varying height (full and pack mode) on the genome. Both are text files that are easy to create, but need to be converted for actual use by the genome browser. The bedGraph format is a very similar format for sparse data or data that contains elements of varying size. bedGraph can also be converted to compressed/indexed binary bigWig files. If you have other data to show in addition to the quantitative data, e.g. data you want to show on mouseover or when the user clicks the feature (like GWAS data), you should have a look at the bigLolly format. For a list of all possible formats for graphing, see the following wiki page. Text files in wiggle format can be uploaded as custom tracks as-is to UCSC where they are compressed and stored for some time. But we recommand that you convert them on your own computer to the binary bigWig storage format. You then copy bigWig files onto your own webserver and they are referenced in custom tracks or track hubs via their URL. Unlike bigWig binar files, wiggle ASCII text files can be uploaded as custom tracks onto our server. After the upload, wiggle data is compressed and stored internally in 128 unique bins. This compression means that there is a minor loss of precision when data is exported from a wiggle track (i.e., with output format "data points" or "bed format" within the Table Browser). For custom tracks, use the bedGraph format if it is important to retain exact data when exporting. However, the size of all custom tracks is limited. For these reasons, we recommend always converting wiggle files to the bigWig storage format and reference these from your custom tracks or track hubs via their URL. General structure Wiggle format is line-oriented. For wiggle custom tracks, the first line must be a track definition line (i.e., track type=wiggle_0), which designates the track as a wiggle track and adds a number of options for controlling the default display. For conversion to bigWig, the most common use case, this line must not be present. Wiggle format is composed of declaration lines and data lines, and require a separate wiggle track definition line. There are two options for formatting wiggle data: variableStep and fixedStep. These formats were developed to allow the file to be written as compactly as possible. variableStep format This format is used for data with irregular intervals between new data points, and is the more commonly used wiggle format. After the wiggle track definition line, variableStep begins with a declaration line and is followed by two columns containing chromosome positions and data values: variableStep chrom=chrN [span=windowSize] chromStartA dataValueA chromStartB dataValueB ... etc ... ... etc ... The declaration line starts with the word variableStep and is followed by a specification for a chromosome. The optional span parameter (default: span=1) allows data composed of contiguous runs of bases with the same data value to be specified more succinctly. The span begins at each chromosome position specified and indicates the number of bases that data value should cover. For example, this variableStep specification: variableStep chrom=chr2 300701 12.5 300702 12.5 300703 12.5 300704 12.5 300705 12.5 is equivalent to: variableStep chrom=chr2 span=5 300701 12.5 Both versions display a value of 12.5 at position 300701-300705 on chromosome 2. Caution about sparse variableStep data: The wiggle format was designed for quickly displaying data that is quite dense. The variableStep format, in particular, becomes very inefficient when there are only a few data points per 1,024 bases. If variableStep data points (i.e., chromStarts) are greater than about 100 bases apart, it is advisable to use BedGraph format. fixedStep format This format is used for data with regular intervals between new data values and is the more compact wiggle format. After the wiggle track definition line, fixedStep begins with a declaration line and is followed by a single column of data values: fixedStep chrom=chrN start=position step=stepInterval [span=windowSize] dataValue1 dataValue2 ... etc ... The declaration line starts with the word fixedStep and includes specifications for chromosome, start coordinate, and step size. The span specification has the same meaning as in variableStep format. For example, this fixedStep specification: fixedStep chrom=chr3 start=400601 step=100 11 22 33 displays the values 11, 22, and 33 as single-base regions on chromosome 3 at positions 400601, 400701, and 400801, respectively. Adding span=5 to the declaration line: fixedStep chrom=chr3 start=400601 step=100 span=5 11 22 33 causes the values 11, 22, and 33 to be displayed as 5-base regions on chromosome 3 at positions 400601-400605, 400701-400705, and 400801-400805, respectively. Note that for both variableStep and fixedStep formats, the same span must be used throughout the dataset. If no span is specified, the default span of 1 is used. As the name suggests, fixedStep wiggles require the same size step throughout the dataset. If not specified, a step size of 1 is used. Data values Wiggle track data values can be integer or real, positive or negative values. Only positions specified have data. Positions not specified do not have data and will not be graphed. All positions specified in the input data must be in numerical order. NaN values are not supported by the browser and, if included, may have unforeseen effects. 1-start coordinate system in use for variableStep and fixedStep The bedGraph format, like all BED-based formats and most file formats used by UCSC, use "0-start, half-open" coordinates, but the wiggle ASCII text format for variableStep and fixedStep data uses "1-start, fully-closed" coordinates. Wiggle (variableStep and fixedStep) is the only format defined by UCSC that uses a 1-based format, for historical reasons. For example, for a chromosome of length N, the first position is 1 and the last position is N. For more information, see: - BigWig and BigBed: enabling browsing of large distributed datasets (Bioinformatics) - Database/browser start coordinates differ by 1 base (Genome Browser FAQ) - The UCSC Genome Browser Coordinate Counting Systems (Genome Browser blog) Parameters for custom wiggle track definition lines All options are placed in a single line separated by spaces (line breaks are inserted in the following example to facilitate readability): track type=wiggle_0 name=track_label description=center_label visibility=display_mode color=r,g,b altColor=r,g,b priority=priority autoScale=on|off alwaysZero=on|off gridDefault=on|off maxHeightPixels=max:default:min graphType=bar|points viewLimits=lower:upper yLineMark=real-value yLineOnOff=on|off windowingFunction=mean+whiskers|maximum|mean|minimum smoothingWindow=off|2-16 The track type with version is REQUIRED, and it currently must be wiggle_0: type wiggle_0 The remaining values are OPTIONAL: name <trackLabel> # default is "User Track" description <centerLabel> # default is "User Supplied Track" visibility <full|dense|hide> # default is hide color <RRR,GGG,BBB> # default is 255,255,255 altColor <RRR,GGG,BBB> # default is 128,128,128 priority <N> # default is 100 For a list of additional options, see the bigWig format page. Note that bigWig files created from bedGraph cannot be converted to wiggle using bigWigToWig and instead will be reverted back to their original bedGraph format. Also, for tracks using altColor with the windowing function "mean+whiskers" the shading of colors will be impacted with lighter shades for values within a standard deviation around the mean, most noticeable when zoomed out and average calculations are taking place. Examples This example specifies 19 separate data points in two tracks (the first is name="variableStep" and the second is name="fixedStep" where priority= orders them) in the region chr19:49,304,200-49,310,700. To view this example as a custom track in the Genome Browser, copy the text and paste it into the "Add Custom Tracks" text box. browser position chr19:49304200-49310700 browser hide all # 150 base wide bar graph at arbitrarily spaced positions, # threshold line drawn at y=11.76 # autoScale off viewing range set to [0:25] # priority = 10 positions this as the first graph # Note, one-relative coordinate system in use for this format track type=wiggle_0 name="variableStep" description="variableStep format" visibility=full autoScale=off viewLimits=0.0:25.0 color=50,150,255 yLineMark=11.76 yLineOnOff=on priority=10 variableStep chrom=chr19 span=150 49304701 10.0 49304901 12.5 49305401 15.0 49305601 17.5 49305901 20.0 49306081 17.5 49306301 15.0 49306691 12.5 49307871 10.0 # 200 base wide points graph at every 300 bases, 50 pixel high graph # autoScale off and viewing range set to [0:1000] # priority = 20 positions this as the second graph # Note, one-relative coordinate system in use for this format track type=wiggle_0 name="fixedStep" description="fixedStep format" visibility=full autoScale=off viewLimits=0:1000 color=0,200,100 maxHeightPixels=100:50:20 graphType=points priority=20 fixedStep chrom=chr19 start=49307401 step=300 span=200 1000 900 800 700 600 500 400 300 200 100 
/goldenPath/help/phastCons.html:Genome_Browser_phastCons_Format	phastCons File Format phastCons data files contain the compressed conservation scores that underlie the Conservation annotation track and the phastCons table. For a detailed description of the algorithm used to produce the scores, see the Genome Browser description page associated with the Conservation track. File Format (assemblies released Nov. 2004 and later) When uncompressed, the file contains a declaration line and one column of data in wiggle table fixed-step format: fixedStep chrom=scaffold_1 start=3462 step=1 0.0978 0.1588 0.1919 0.1948 0.1684 The declaration line specifies the starting point of the data in the assembly. It consists of the following fields: - fixedStep -- keyword indicating the wiggle track format used to write the data. In fixed step format, the data is single-column with a fixed interval between values. - chrom -- chromosome or scaffold on which first value is located. - start -- position of first value on chromosome or scaffold specified by chrom. NOTE: Unlike most Genome Browser coordinates, these are one-based. - step -- size of the interval (in bases) between values. A new declaration line is inserted in the file when the chrom value changes, when a gap is encountered (requiring a new start value), or when the step interval changes. Data lines The first data value below the header shows the score corresponding to the position specified in the header. Subsequent score values step along the assembly in one-base intervals. The score shows the posterior probability that the phylogenetic hidden Markov model (HMM) of phastCons is in its most-conserved state at that base position. ------------------------------------------------------------------------ File Format (assemblies prior to Nov. 2004) When uncompressed, the data file contains two columns: 294 0.0953 295 0.0948 296 0.0943 297 0.0936 298 0.0929 299 0.0921 Column #1 contains a one-based position coordinate. Column #2 contains a score showing the posterior probability that the phylogenetic hidden Markov model (HMM) of phastCons is in its most conserved state at that base position. ------------------------------------------------------------------------ References for phastCons Siepel A and Haussler D (2005). Phylogenetic hidden Markov models. In R. Nielsen, ed., Statistical Methods in Molecular Evolution, pp. 325-351, Springer, New York. Siepel, A., Bejerano, G., Pedersen, J.S., Hinrichs, A., Hou, M., Rosenbloom, K., Clawson, H., Spieth, J., Hillier, L.W., Richards, S., Weinstock, G.M., Wilson, R. K., Gibbs, R.A., Kent, W.J., Miller, W., and Haussler, D. Evolutionarily conserved elements in vertebrate, insect, worm, and yeast genomes. Genome Res. 15, 1034-1050 (2005). For a discussion of the methods used to calculate the phastCons scores, see the description page for the hg17 Conservation track in the Genome Browser. 
/goldenPath/help/fileSearch.html:Genome_Browser_File_Search	File Search The Genome Browser's File Search feature allows users to find downloadable ENCODE files of interest quickly and easily. Searching Downloadable ENCODE files can be found by entering terms of interest in the free-text Track Name and Description fields, by selecting the appropriate Group or Data Format from the drop-down menus, and/or by using the "ENCODE terms" drop-down filters. ENCODE terms: The filters in this section allow users to search (or refine their search) for files based on the metadata associated with the ENCODE downloadable files. By default, there are two rows of ENCODE metadata search criteria, but more can be added by clicking the "+" button. To remove an added metadata search row, click the "-" button. Each row is comprised of two menus. The first is a drop-down menu that contains the searchable metadata categories. Please note that because the metadata categories are assembly dependent, the options may vary from assembly to assembly.Some metadata categories have a link on the far right of the row that provides more information about that category. The second menu is usually a multi-select drop-down menu containing the the metadata terms available for the category selected (in the first drop-down). The default is usually set to "Any". Clicking on "Any" opens up the multi-select drop-down with the possible options. Multiple options can be selected by clicking multiple checkboxes. For a few of the metadata categories in the first drop-down menu, the second menu is a free-text field in which the desired term must be entered. Results After clicking the "Search" button, the files that meet the search criteria (up to a maximum of 1000 files) are displayed in a table, one file per row. If the displayed search results have been limited to the first 1000 files, a message stating this restriction will be displayed above the results. Results Header: The first column of the header contains the number of files found (up to a maximum of 1000 files) and the subsequent columns contain a title for the information displayed in that column. The list of files can be sorted by clicking on the header of the column. The files can be sorted on up to 5 attributes. Files: Click the "Download" button to download a file. Click the [] icon to go to the downloads page to view that file and all other files associated with the track. 
/goldenPath/help/mysql.html:Genome_Browser_MySql_Downloads	Downloading Data using MariaDB (MySQL) The UCSC Genome Browser uses MariaDB as the backend database server. MariaDB is a community-developed, commercially supported fork of the MySQL relational database management system, intended to remain free and open-source software under the GNU General Public License. We have two MariaDB databases for public access: - genome-mysql.soe.ucsc.edu (located on the US west coast) - genome-euro-mysql.soe.ucsc.edu (located in Europe) These servers allow MySQL access to the same set of data currently available on our public Genome Browser site. The data are synchronized weekly with the main databases on our public site. During synchronization, the MariaDB server can be intermittently out of sync with the main website for a short period of time. The weekly synchronization takes place on Monday mornings from 4:00 am to 9:00 am Pacific Time (GMT -7:00 in summer, GMT -8:00 in winter). Connecting You must have MariaDb (MySQL) client libraries installed on your computer. You can read more about MariaDB on the MariaDB site. You can connect to the US MariaDB server using the command: mysql --user=genome --host=genome-mysql.soe.ucsc.edu -A -P 3306 Or the European MariaDB server with this command: mysql --user=genome --host=genome-euro-mysql.soe.ucsc.edu -A -P 3306 The -A flag is optional but is recommended for speed. Once connected to the database, you may use a wide range of SQL commands to query the database. Conditions of use - Avoid excessive or heavy queries that may impact the server performance. Inappropriate query use will result in a restriction of access. If you plan to execute a query that you think may be excessive, contact UCSC first to avoid the possibility of having your access blocked. - Bot access and excessive program-driven use are not permitted. - Attachments by local mirror sites are prohibited. For more details about the Conditions of Use, please refer to the following page, Genome Browser Conditions of Use. Using the MariaDB server with our utilities The MariaDB database can also be used by the numerous utilities in the Genome Browser source tree. Some of these utilities require a password, so you will need to add the following specifications to your $HOME/.hg.conf file (remember to chmod your .hg.conf file to 600 permissions) if you would like to access the US public MariaDB server: #US MariaDB server db.host=genome-mysql.soe.ucsc.edu db.user=genomep db.password=password central.db=hgcentral central.host=genome-mysql.soe.ucsc.edu central.user=genomep central.password=password gbdbLoc1=http://hgdownload.soe.ucsc.edu/gbdb/ forceTwoBit=on Or these lines if you'd like to access the European MariaDB server: #European MariaDB server db.host=genome-euro-mysql.soe.ucsc.edu db.user=genomep db.password=password central.db=hgcentral central.host=genome-euro-mysql.soe.ucsc.edu central.user=genomep central.password=password gbdbLoc1=http://hgdownload.soe.ucsc.edu/gbdb/ forceTwoBit=on The db.* and central.* settings tell our utilities how to connect to the public MariaDB server. The gbdbLoc1 setting tells our utilities where to find data files. The forceTwoBit setting is necessary for utilities that retrieve genomic sequence. If you have set up your .hg.conf file as above, you can use the hgsql utility, available from our downloads server in the Utilities section, to access the public MariaDB server. The benefit of using the hgsql command is that you don't have to include the username or password as part of your command. You only need to specify the host: hgsql -h genome-mysql.soe.ucsc.edu If you prefer a graphical user interface (GUI) to the UCSC database tables, use the Table Browser. If you would like to learn more about .hg.conf file setup and specifics for using our command-line utilities, see this example minimal.hg.conf file. System problems should be reported to genome-www@soe.ucsc.edu. Send questions regarding the database contents or queries to genome@soe.ucsc.edu. Messages sent to this address will be posted to the moderated genome mailing list, which is archived on a SEARCHABLE, PUBLIC Google Groups forum. 
/goldenPath/help/covidBrowser.html:COVID_Browser_Intro	Introduction to the SARS-CoV-2 Genome Browser The UCSC Genome Browser is an open-source, interactive sequence visualization tool that has been a cornerstone of genomics since we released the first human genome assembly 20 years ago. Cited in more than 37,000 scientific articles and used by thousands of researchers each day; it allows for cross-referencing of research, clinical, and epidemiology data against reference genomes, including SARS-CoV-2. This data is continuously updated and added to as new datasets become available. For a more thorough description, please reference our SARS-CoV-2 Genome Browser Nature Genetics paper. We also post updates and COVID Browser resources to out COVID-19 Browser home page. This guide will go through some of the most important use cases of the SARS-CoV-2 Genome Browser. These topics include: - Orientation and Navigation - Gene Data and Sequence Alignments - Variation and Immunology data - Phylogenetic Contact Tracing using USHER - Other tools and data downloads - Support and Collaboration For those who prefer a video explanation, we also have the following tutorial: []       SARS-CoV2 Browser Tutorial Video - for other videos, see our Video Page. Visit our YouTube channel for more videos. Genome Browser Orientation and Navigation The standardized reference genome displayed on the COVID Genome Browser is from one of the first isolated cases, known as NC_045512v2 or wuhCor1. With more than 80 track datasets across the SARS-CoV-2 reference genome's nearly 30,000 RNA bases, navigation is essential to finding the information you want to see. Below is an example view of the SARS-CoV-2 Genome Browser with labeled sections highlighting the navigation, reference sequence, annotations, and other available track datasets. Navigation controls at the top allow users to move left and right and to zoom. The search box allows users to search for particular features or to move to exact genomic coordinates. The RNA sequence is shown at the top only when the view is sufficiently zoomed in. Annotations are shown for data tracks that have been set to visible in the available tracks section at the bottom. Tracks can be configured with a right-click or by clicking on their name near the bottom of the page. [Labeled orientation to the Genome Browser] This is a view of the SARS-CoV-2 Genome Browser (COVID Browser) with labeled elements to help with orientation. Interact with this session by clicking on the picture. To read the full caption, please go to our Nature Genetics paper. Genes and Sequence Alignments Gene and protein annotations are organized by the contributor, most notably NCBI and UniProt. Having multiple information sources allows a consensus to be formed among datasets. Like many viral genomes, molecular complexity arises from polyproteins rearranging, generating ~29 protein products. Most notable among these is the S (spike) protein which defines coronaviruses and allows entry into cell membrane. Additional tracks contain information such as interactions between viral proteins and human proteins (protein interact), PDB structures, and RNA structure annotations (Rangan RNA), and more. Sequence alignments and conservation data are also available across the SARS-CoV-2 genome, from large-scale views to individual bases and amino acids. Four conservation tracks compare sequences with 44 bat coronaviruses, 119 vertebrate coronaviruses, 7 human coronaviruses, and PhyloCSF computed conservation scores. The tracks display differently depending on visibility mode and the number of bases on the screen. Datasets can be turned on by setting the dropdown next to the data track name from "hide" to dense, squish, pack, or full. Then click the refresh button to see these changes in effect. Clicking on a data track name will take you to a description with more information on the dataset, display conventions, methods, and references. Clicking on a particular item will take you to a page with complete information about that item and dataset. [Some of the gene and conservation data on the Genome Browser] This Genome Browser display shows some of the gene and conservation tracks available on the SARS-CoV-2 genome. You should be able to see UniProt protein products, regions of interest, and domains all mapped against the SARS-CoV-2 genome. Below those tracks are two different conservation alignments in "squish" and "pack" formats, comparing bat-host and human-host coronavirus sequences with the reference SARS-CoV-2 genome. Interact with this session by clicking on the picture. Exploring Variation and Immunology Data The SARS-CoV-2 Genome Browser displays data on variation within SARS-CoV-2 from UniProt, GenBank, GISAID, Nexstrain, and other providers. These datasets cover global trends in SARS-CoV-2 variation among all available public sequences, with regional descriptions available through clicking into a particular entry. A few of the most notable tracks under the "Variation and Repeats" section are the Phylogeny: Public track, which shows a continuously updating phylogenetic tree that clusters similar sequences, with the frequency of each mutation shown by the height of the bar at that particular base. Tools are provided to filter these data to show only well-supported mutation calls, set thresholds for minor-allele frequency, and display data for specific clades. Another track is the spike protein mutations from community annotations, highlighted as amino acid changes with red indicating strong antibody escape in receptor-binding domain (RBD) mutation screens. The Genome Browser has also has the Variants of Concern track, which pinpoints each accumulated mutation that defines 4 strains of SARS-CoV-2 of particular concern, labeled based on lay terms (such as 'California variant') as well as the using the lineage defined by the Pangolin software (such as 'B.1.1.7'). The Genome Browser also provides 12 immunology datasets that can inform potential therapeutic targets or public health risks. Protein epitopes are highlighted in the genome by multiple tracks, including those from the Immune Epitope Database (IEDB) and from a study of COVID+ patients. Of particular interest are the datasets describing surveys of antibody response across a variety of SARS-CoV-2 variants in the receptor-binding domain (Antibody Escape Mutations). [Some of the variation and immunology data on the Genome Browser] This image shows some of the variation data tracks that can be displayed on the SARS-CoV-2 genome, specifically zoomed into the receptor-binding domain of the Spike protein. Validated epitopes are displayed in black that may be a target for therapeutic antibodies. In red and black, antibody escape scores are are shown for each genome position. Smaller tick marks show amino acid or nucleotide changes from different sources, with more information available by clicking into the item. Genetic Contact Tracing with UShER The UCSC Genome Browser has developed a tool that allows placement of SARS-CoV-2 sequences onto existing phylogenetic trees far faster than previous methods, allowing instantaneous tracing of strains and transmission events. This tool is called Ultrafast Sample placement on Existing tRees (UShER) and exists as an interactive web-tool to compare sequences and link to existing public phylogenetic trees. [Example of the UShER phylogeny placement tool] After uploading a Fasta file, the tool returns a page with quality metrics such as: number of bases aligned, number of Ns, and number of maximally parsimonious placements along with the lineage and clade of the nearest neighbor. Colored boxes highlight possible quality issues, green meaning this was a high confidence placement. SARS-CoV-2/ COVID Phylogenic Trees You can view your aligned SARS-CoV-2 sequence genotypes along with their closest known relatives among the 150,000+ public sequences. You can compare among your uploaded samples or trace possible transmission vectors using mutational signatures. [Example of the UShER phylogeny placement tool tree features] The uploaded sequences are highlighted in blue alongside their most closely aligned public sequences. You can investigate genotypes and relationships between samples. Other tools, downloads, and features Custom Tracks, BLAT, Track Hubs Along with a suite of data tracks, filters, and visualization options for the SARS-CoV-2 genome, the UCSC Genome Browser offers many additional ways to interface with our data. You can upload your data on the reference genome in nearly any format with our Custom Track tool. If you have unaligned sequence, you can use our BLAT sequence alignment tool to get coordinates and base-by-base comparison with any reference genome. We also display formatted data as Track Hubs and curate a list of user-submitted Public Track Hubs. Downloads, Table Browser, JSON API, SQL As part of our open-source, open-access philosophy, we try to make it as easy as possible for researchers to download entire datasets or filtered subsets. Each track description page has a Data Access section which points users to our main options for data download. For downloading complete datasets, our SARS-CoV-2 download directory provides access to all our source files for transparency and reproducibility. Our Table Browser tool lets users interact with our data using a variety of filters based on score, identifiers, or any other field. Table Browser also allows users to convert data into multiple different formats (e.g. BED, GTF) and to access different formatted sequence outputs (in FASTA format). We have a JSON API which can be programmatically called and return any dataset in its entirety or as a filtered subset based on documented input parameter. We also offer a Public SQL server for similar flexible, automatic way to access genomic data and annotations. Along with this particular virus genome browser, we have thousands of genomes available for visualization and analysis from our genome assemblies gateway page. Support and Collaboration The Genome Browser offers rapid email support for anything related to our tools. If your question is general or may have been asked before, please review our Browser documentation and our archive of previously answered questions. If you would still like help, please go to our Contact Us page to see access our email support. When contacting us, please include a session link, images, and example data if applicable. We are active on social media, you can follow us on Twitter or Facebook. We are always looking to collaborate with researchers and add new datasets to our site. We also seek to continuously improve our tools to meet the needs of the scientific community. If you have any collaboration ideas, contributions, or feature requests, please reach out through our suggestion page. 
/goldenPath/help/hgBaseLabel.html:Genome_Browser_Base_Coloring	Base Coloring for Alignment Tracks The Genome Browser's codon coloring feature allows users to quickly validate and compare alignment tracks, such as ESTs or BLAT results. To turn on codon coloring, select the desired option from the Color track by bases pull-down menu. Color Options - item bases - all bases are labeled. - different item bases - red: only those bases that differ from the genomic sequence are labeled or colored red, depending on the zoom level. Details Bases are colored or labeled according to the genomic sequence. Note that it is possible to show the complement strand, and thus change the base labeling, by clicking the arrow to the left of the base display in the base position track. When zoomed out past the base level, the browser will choose one color to represent many bases. The priority of display, from most important to least important, is: different item bases (if enabled) and then alignment coloring (if enabled). To view labeling, the track must be zoomed to within 3 times the base level. For information about alignment insertion/deletion display options click here. 
/goldenPath/help/blatSpec.html:Blat_Spec_and_User's_Guide	Blat Suite Program Specifications and User Guide Blat produces two major classes of alignments: - at the DNA level between two sequences that are of 95% or greater identity, but which may include large inserts. - at the protein or translated DNA level between sequences that are of 80% or greater identity and may also include large inserts. The output of blat is flexible. By default, it is a simple tab-delimited file that describes the alignment, but which does not include the sequence of the alignment itself. Alternatively, blat can produce output compatible with BLAST or WU-BLAST, as well as several other formats. The main programs in the blat suite are: - gfServer – a server that maintains an index of the genome in memory and uses the index to quickly find regions with high levels of sequence similarity to a query sequence. - gfClient – a program that queries gfServer over the network and does a detailed alignment of the query sequence with regions found by gfServer. - blat – client and server combined into a single program, first building the index, then using the index, and then exiting. - webBlat – a web-based version of gfClient that presents the alignments in an interactive fashion. Building an index of the genome typically takes 10 or 15 minutes. For interactive applications, it is typical to use gfServer to build a whole-genome index. gfClient or webBlat can then align a single query within a few seconds. When aligning a large number of sequences in a batch mode, it may be more efficient to use blat, particularly if it is run on a cluster of computers. Each blat run is typically done against a single chromosome, but with a large number of query sequences. Other programs in the blat suite are: - pslSort – combines and sorts the output of multiple blat runs. The blat default output format is psl. - pslReps – selects the best alignments for a particular query sequence, using a "near best in genome" approach. - pslPretty – converts alignments from psl format, which is a tab-delimited format that does not include the bases themselves, to a more readable alignment format. - faToTwoBit – converts fasta format sequence files to the dense, randomly accessible .2bit format that gfClient can use. - twoBitToFa – converts from .2bit format back to fasta format. - faToNib – converts from fasta to the somewhat less dense, randomly accessible .nib format that predated .2bit format. Note that each .nib file can contain only a single sequence. - nibFrag – converts portions of a .nib file back to fasta format. Usage details for each of the above programs are described in the sections below. For all programs except webBlat, the usage options may also be viewed by typing the name of the program (without arguments) on the UNIX command line. The gfServer program requires approximately 1 byte of memory for every 3 bases in the genome it is indexing in DNA mode, and 1.5 bytes for each unmasked base in translated mode. The blat program requires approximately 2 bytes of memory for each base in the genome in DNA mode, and 3 bytes of memory for each base in translated mode. The other programs use relatively little memory. You may also be interested in the following programs that are not part of the blat suite: - in-silico PCR – quickly finds the sequence between two primers. This includes webPCR, an interface similar to webBlat. - Genome Browser – displays annotations as a series of tracks on top of the genome. All of the above software and source code are freely available from the University of California Santa Cruz for non-profit academic use. A license is required for commercial use. For blat licensing, contact Kent Informatics. All other licensing is handled by the UCSC Genome Browser project. blat blat – Standalone blat sequence search command line tool. Usage: blat database query [-ooc=11.ooc] output.psl Where: database and query are each either a .fa, .nib, or .2bit file, or a list of these files, one file name per line. -ooc=11.ooc tells the program to load over-occurring 11-mers from an external file. This will increase the speed by a factor of 40 in many cases, but is not required. output.psl is the output file. Subranges of .nib and .2bit files may be specified using the syntax: /path/file.nib:seqid:start-end or /path/file.2bit:seqid:start-end or /path/file.nib:start-end With the second form, a sequence id of file:start-end is used. Options: -t=type Database type, one of: dna - (default) DNA sequence prot - protein sequence dnax - DNA sequence translated in six frames to protein -q=type Query type, one of: dna - DNA sequence rna - RNA sequence prot - protein sequence dnax - DNA sequence translated in six frames to protein rnax - DNA sequence translated in three frames to protein -prot Synonymous with -t=prot -q=prot -ooc=N.ooc Use overused tile file N.ooc. N should correspond to the tileSize. -tileSize=N Sets the size of match that triggers an alignment. Usually between 8 and 12. Default is 11 for DNA and 5 for protein. -stepSize=N Spacing between tiles. Default is tileSize. -oneOff=N If set to 1, this allows one mismatch in tile and still triggers an alignment. Default is 0. -minMatch=N Sets the number of tile matches. Usually set from 2 to 4. Default is 2 for nucleotide, 1 for protein. -minScore=N Sets minimum score. This is the matches minus the mismatches minus some sort of gap penalty. Default is 30. -minIdentity=N Sets minimum sequence identity (in percent). Default is 90 for nucleotide searches, 25 for protein or translated protein searches. -maxGap=N Sets the size of maximum gap between tiles in a clump. Usually set from 0 to 3. Default is 2. Relevant only for minMatch > 1. -noHead Suppresses .psl header (so it's just a tab-separated file). -makeOoc=N.ooc Makes overused tile file. Target must be complete genome. -repMatch=N Sets the number of repetitions of a tile allowed before it is marked as overused. Typically this is 256 for tileSize 12, 1024 for tile size 11, 4096 for tile size 10. Default is 1024. Typically needed only with makeOoc. Also affected by stepSize: when stepSize is halved, repMatch is doubled to compensate. -noSimpRepMask Suppresses simple repeat masking. -mask=type Masks out repeats. Alignments won't be started in masked region but may extend through it in nucleotide searches. Masked areas are ignored entirely in protein or translated searches. Types are: lower - masks out lower-cased sequence upper - masks out upper-cased sequence out - masks according to database.out RepeatMasker .out file file.out - masks database according to RepeatMasker file.out -qMask=type Masks out repeats in query sequence. Similar to -mask above but for query rather than target sequence. -repeats=type Type is same as mask types above. Repeat bases will not be masked in any way, but matches in repeat areas will be reported separately from matches in other areas in the psl output. -minRepDivergence=NN Minimum percent divergence of repeats to allow them to be unmasked. Default is 15. Relevant only for masking using RepeatMasker .out files. -dots=N Outputs a dot every N sequences to show program's progress. -trimT Trims leading poly-T. -noTrimA Don't trim trailing poly-A. -trimHardA Removes poly-A tail from qSize as well as alignments in psl output. -fastMap Run for fast DNA/DNA remapping, not allowing introns and requiring high %ID. -out=type Controls output file format, one of: psl - (Default) tab-separated format, no sequence pslx - tab separated format with sequence axt - blastz-associated axt format maf - multiz-associated maf format sim4 - similar to sim4 format wublast - similar to wublast format blast - similar to NCBI blast format blast8 - NCBI blast tabular format blast9 - NCBI blast tabular format with comments -fine For high-quality mRNAs, looks harder for small initial and terminal exons. Not recommended for ESTs. -maxIntron=N Sets maximum intron size. Default is 750000. -extendThroughN Allows extension of alignment through large blocks of Ns. Blat settings for common usage scenarios: Mapping ESTs to the genome within the same species: -ooc=11.ooc Mapping full length mRNAs to the genome in the same species: -ooc=11.ooc -fine -q=rna Mapping ESTs to the genome across species: -q=dnax -t=dnax Mapping mRNA to the genome across species: -q=rnax -t=dnax Mapping proteins to the genome: -q=prot -t=dnax Mapping DNA to DNA in the same species: -ooc=11.ooc -fastMap Mapping DNA from one species to another species: -q=dnax -t=dnax Note that the query side of the alignment should be split into chunks of 25 KB or less for best performance. gfServer gfServer – Make a server to quickly find where DNA occurs in genome. Usage: To set up a server: gfServer start host port file(s) where files are in .nib or .2bit format To remove a server: gfServer stop host port To query a server with DNA sequence: gfServer query host port probe.fa To query a server with protein sequence: gfServer protQuery host port probe.fa To query a server with translated DNA sequence: gfServer transQuery host port probe.fa To query server with PCR primers gfServer pcr host port fPrimer rPrimer maxDistance To process 1 probe fa file against .nib format genome (not starting server): gfServer direct probe.fa file(s).nib To test pcr without starting server: gfServer pcrDirect fPrimer rPrimer file(s).nib To figure out usage level: gfServer status host port To get input file list: gfServer files host port Options: -tileSize=N Size of n-mers to index. Default is 11 for nucleotides, 4 for proteins (or translated nucleotides). -stepSize=N Spacing between tiles. Default is tileSize. -minMatch=N Number of n-mer matches that triggers detailed alignment. Default is 2 for nucleotides, 3 for proteins. -maxGap=N Number of insertions or deletions allowed between n-mers. Default is 2 for nucleotides, 0 for proteins. -trans Translate database to protein in 6 frames. Note: when using this option, it is best to run on RepeatMasked data. -log=logFile Keep a log file that records server requests. -seqLog Include sequences in log file (not logged with -syslog). -syslog Log to syslog. -logFacility=facility Log to the specified syslog facility. Default is local0. -mask Use masking from .nib file. -repMatch=N Number of occurrences of a tile (n-mer) that triggers repeat-masking of the tile. Default is 1024. -maxDnaHits=N Maximum number of hits sent from the server for a DNA query. Default is 100. -maxTransHits=N Maximum number of hits sent from the server for a translated query. Default is 200. -maxNtSize=N Maximum size of untranslated DNA query sequence. Default is 40000. -maxAaSize=N Maximum size of protein or translated DNA queries. Default is 8000. -canStop If set, a quit message will actually take down the server. gfClient gfClient – A client for the genomic finding program that produces a .psl file. Usage: gfClient host port seqDir in.fa out.psl Where: host is the name of the machine running the gfServer. port is the same port used for the gfServer. seqDir is the path of the .nib or .2bit files relative to the current dir. Note: these are needed by the client as well as the server. in.fa is a fasta format file. May contain multiple records. out.psl is the output file. Options: -t=type Database type, one of: dna - (Default) DNA sequence prot - protein sequence dnax - DNA sequence translated in six frames to protein -q=type Query type, one of: dna - DNA sequence rna - RNA sequence prot - protein sequence dnax - DNA sequence translated in six frames to protein rnax - DNA sequence translated in three frames to protein -prot Synonymous with -t=prot -q=prot -dots=N Outputs a dot every N query sequences. -nohead Suppresses psl header. -minScore=N Sets minimum score. This is twice the matches minus the mismatches and minus a gap penalty. Default is 30. -minIdentity=N Sets minimum sequence identity (in percent). Default is 90 for nucleotide searches, 25 for protein or translated protein searches. -out=type Controls output file format, one of: psl - (Default) tab-separated format without actual sequence pslx - tab-separated format with sequence axt - blastz-associated axt format maf - multiz-associated maf format sim4 - similar to sim4 format wublast - similar to wublast format blast - similar to NCBI blast format blast8- NCBI blast tabular format blast9 - NCBI blast tabular format with comments -maxIntron=N Sets maximum intron size. Default is 750000. faToTwoBit faToTwoBit – Convert DNA from fasta to .2bit format. Usage: faToTwoBit in.fa [in2.fa in3.fa ...] out.2bit Options: -noMask Ignore lower-case masking in fa file. -stripVersion Strip off version number after "." for GenBank accessions. -ignoreDups Convert only first sequence if there are duplicate sequence names. Use "twoBitDup" to find duplicate sequences. twoBitToFa twoBitToFa – Convert all or part of .2bit file to fasta format. Usage: twoBitToFa input.2bit output.fa Options: -seq=name Restrict this to just one sequence. -start=X Start at given position in sequence (zero-based). -end=X End at given position in sequence (non-inclusive). -seqList=file File containing list of the desired sequence names in the format seqSpec[:start-end], e.g. chr1 or chr1:0-189, where coordinates are half-open zero-based, i.e. [start,end). -noMask Convert sequence to all upper-case. -bpt=index.bpt Use bpt index instead of the built-in one. -bed=input.bed Grab sequences specified by input.bed. Will exclude introns. -bedPos With -bed, to use chrom:start-end as the fasta ID in output.fa. -udcDir=/dir/to/cache Location of cache for remote bigBeds and bigwigs. Sequence and range may also be specified as part of the input file name using the syntax: /path/input.2bit:name or /path/input.2bit:name or /path/input.2bit:name:start-end faToNib faToNib – Convert from .fa to .nib format. Usage: faToNib [options] in.fa out.nib Options: -softMask Create nib that soft-masks lower-case sequence. -hardMask Create nib that hard-masks lower-case sequence. nibFrag nibFrag – Extract part of a .nib file as .fa, with all bases and gaps lower-case by default. Usage: nibFrag [options] file.nib start end strand out.fa Where: strand is + (plus) or m (minus) Options: -masked Use lower-case characters for bases meant to be masked out. -hardMasked Use upper-case characters for bases that are not masked out, and Ns for masked-out bases. -upper Use upper-case characters for all bases. -name=name Use given name after '>' in output sequence. -dbHeader=db Add full database info to the header, with or without -name option. -tbaHeader=db Format header for compatibility with tba, takes database name as argument. pslPretty pslPretty – Convert PSL to human-readable output Usage: pslPretty in.psl target.lst query.lst pretty.out Options: -axt Save in axt format. -dot=N Output a dot every N records. -long Don't abbreviate long inserts. -check=fileName Output alignment checks to filename. To improve performance, a .psl file containing multiple targets should be sorted by target. The target and query lists can be .fasta, .2bit, or .nib files, or a list of these files, one per line. pslSort pslSort – Merge and sort psCluster .psl output files. Usage: To sort all of the .psl files in the "inDirs" directories in two stages: first into temporary files in tempDir, and then into outFile: pslSort dirs[1|2] outFile tempDir inDir(s) The device on tempDir must have sufficient space, typically 15-20 gigabytes if processing a whole genome. To sort a genome-to-genome alignment, reflecting the alignments across the diagonal: pslSort g2g[1|2] outFile tempDir inDir(s) Appending [1|2] to the "dirs" or "g2g" option limits the program to only the first or second pass of the sort. Options: -nohead Suppress the psl header. -verbose=N Set verbosity level, higher for more output. Default is 1. Note: pslSort will run out of memory on extremely large files. It may be preferable to use this UNIX command in these situations: sort -k 10 *.psl > sorted.psl In this case, remove the header lines from the .psl input file using the "-noHead" option to blat before using the UNIX sort command. pslReps pslReps – Analyze repeats and generate genome-wide best alignments from a sorted set of local alignments. Usage: pslReps in.psl out.psl out.psr Where: in.psl is an alignment file generated by psLayout and sorted by pslSort out.psl is the best alignment output out.psr contains repeat info Options: -nohead Suppress psl header. -ignoreSize Reduce weighing in favor of larger alignments. -noIntrons Don't penalize for not having introns when calculating size factor. -singleHit Takes single best hit, not splitting into parts. -minCover=0.N Minimum coverage to output. Default is 0. -ignoreNs Ignore Ns when calculating minCover. -minAli=0.N Minimum alignment ratio. Default is 0.93. -nearTop=0.N Amount of deviation from top allowed to be taken. Default is 0.01. -minNearTopSize=N Minimum size of alignment that is near top for alignment to be kept. Default 30. -coverQSizes=file Tab separate file with effective query sizes. When used with -minCover, this allows polyAs to be excluded from the coverage calculation. Setting up WebBlat Installing webBlat Installing a web-based blat server involves four major steps: 1. Create the sequence databases. 2. Run the gfServer program to create in-memory indexes of the databases. 3. Edit the webBlat.cfg file to set the machine and port that the gfServer(s) are running on, and to optionally customize the webBlat appearance for users. 4. Copy the webBlat executable and webBlat.cfg to a directory where the web server can execute webBlat as a cgi. Creating sequence databases Create databases with the program faToTwoBit. Typically, a separate database is created for each genome being indexed. Each database can contain up to four billion bases of sequence in an unlimited number of records. The databases for webPcr and webBlat are identical. The input to faToTwoBit is one or more fasta format files, each of which can contain multiple records. If the sequence contains repeat sequences, as is the case with vertebrates and many plants, the repeat sequences can be represented in lower-case and the other sequence in upper-case. The gfServer program can then be configured to ignore the repeat sequences. The output of faToTwoBit is a file that is designed for fast random access and efficient storage. The output file stores four bases per byte, and uses a small amount of additional space to store the case of the DNA and to keep track of blocks of Ns in the input. Other ambiguity codes in the input sequence, such as Y and U, will be converted to N. Here is an example of how a typical installation might create a mouse and a human genome database: cd /data/genomes mkdir twoBit faToTwoBit human/hg19/*.fa twoBit/hg19.2bit faToTwoBit mouse/mm10/*.fa twoBit/mm10.2bit It is not necessary to put all of the databases in the same directory, but it can simplify bookkeeping. The databases can use the older blat format, .nib, instead of the .2bit format. The .nib format packs only two bases per byte, and handles only one record per .nib file. Creating in-memory indexes with gfServer The gfServer program creates an in-memory index of a nucleotide sequence database that can be used for translated or untranslated searches. Translated indexes enable protein-based blat queries and use approximately two bytes per unmasked base in the database. Untranslated indexes are used for nucleotide-based blat queries and the in-silico PCR utility. A normal blat index uses approximately one-quarter of a byte per base. For blat on smaller (primer-sized) queries or for in-silico PCR, it is recommended that you use a more thorough index that requires one-half of a byte per base. The gfServer program is memory intensive but typically does not require a lot of CPU power. With sufficient memory, multiple gfServers can be run on the same machine. Here is an example of a typical installation: ssh bigRamMachine cd /data/genomes/twoBit gfServer start bigRamMachine 17779 hg19.2bit & gfServer -trans -mask start bigRamMachine 17778 hg19.2bit & The "trans" flag creates a translated index. It requires approximately 15 minutes to build an untranslated index and 45 minutes to build a translated index. To build an untranslated index to be shared with in-silico PCR: gfServer -stepSize=5 bigRamMachine 17779 hg19.2bit & This index will be slightly more sensitive with blat, particularly for small query sequences. Editing the webBlat.cfg file The webBlat.cfg file tells the webBlat program where to look for gfServers and sequence. The basic format of the .cfg file is line-oriented. The first word of each line is a command, and lines that are blank or start with "#" are ignored. The webBlat.cfg and webPcr.cfg files are similar. The webBlat.cfg commands are: - gfServer – defines the host and port on which the (untranslated) gfServer is running, the associated sequence directory, and the name of the database to display on the web page. - gfServerTrans – defines the location of a translated server. - background – (optional) defines the background image (if any) to display on the web page. - company – (optional) defines the company name to display on the web page. - tempDir – specifies where to put temporary files. This path is relative to the directory where the web server executes CGI scripts. It is recommended that an automated mechanism, such as a cron job, be put in place to periodically remove files that haven't been recently accessed. The background and company commands are optional. The webBlat.cfg file must have at least one valid gfServer or gfServerTrans line, and a tempDir line. Here is an example of a webBlat.cfg file that you might find for a typical webBlat installation: company Awesome Research Amalgamated background /images/dnaPaper.jpg gfServerTrans bigRamMachine 17778 /data/genomes/twoBit/hg19.2bit Human Genome gfServer bigRamMachine 17779 /data/genomes/twoBit/hg19.2bit Human Genome gfServerTrans mouseServer 17780 /data/genomes/twoBit/mm10.2bit Mouse Genome gfServer mouseServer 17781 /data/genomes/twoBit/mm10.2bit Mouse Genome tempDir ../trash Locating webBlat The details of this step vary highly among web servers. Unless you are administering your own computer, you will likely have to ask your local system administrators for help with this part of the webBlat installation. Here is an example of a typical installation of Apache: ssh webServer cd kent/webBlat cp webBlat webBlat.cfg /usr/local/apache/cgi-bin/ mkdir /usr/local/apache/trash chmod 777 /usr/local/apache/trash In the above example, the executable and configuration file reside in the directory kent/webBlat/. The program will create some files in the trash directory. It is advisable to periodically remove old files from this directory. Here is an example of an installation on Mac OS X: cp webBlat webBlat.cfg /Library/WebServer/CGI-Executables/ mkdir /Library/WebServer/trash chmod 777 /Library/WebServer/trash 
/goldenPath/help/hgRegMotifHelp.html:Genome_Browser_Motif_Display	Motif Display Help Page A motif is a predominant regulatory sequence theme associated with a specific transcription factor. This collection of common sites can be represented by a consensus sequence, where a lower-case base, for example, signifies a lower degree of frequency compared to an upper-case base. A more accurate display is a position weight matrix (PWM), which gives the probability of each base numerically. A motif can also be displayed graphically as a sequence logo, where the height of each letter is proportional to its frequency at that particular position, but the total height represents the mutual information content. All three methods convey the same underlying information, numerically displayed in the PWM. [Motif example] Example ETS1 info from hg19 at chr10:69644211-69644221 This Occurrence line: This line represents the assembly's sequence, from the strand the motif matched. This sequence is the only line referring to the underlying assembly. The sequence logo, motif consensus, and numerical table are instead all generated from the underlying PWM. Motif Consensus line: The motif consensus line uses the following scheme to indicate strength of the highest-probability base for each column in the motif: Weight Display Convention Example ----------- -------------------- --------- .90 - 1.0 upper-case bold ACGT .75 - .89 upper-case normal ACGT .50 - .74 lower-case normal acgt .40 - .49 lower-case grayed acgt .00 - .39 lower-case grayed . Position Weight Matrix (PWM): Each row of the PWM represents a base, and each column represents the relative frequency of the base occurring for that position. Sequence Logo: A sequence logo is a stack of letters at each position, where the relative sizes of the letters to each other represents their frequency in the sequence. The total height of the stack of letters reflects the mutual information content at the position, in bits. Resources and references Schneider TD, Stephens RM. Sequence logos: a new way to display consensus sequences. Nucleic Acids Res. 1990 Oct 25;18(20):6097-100. PMID: 2172928; PMC: PMC332411 Stormo GD. DNA binding sites: representation and discovery. Bioinformatics. 2000 Jan;16(1):16-23. PMID: 10812473 
/goldenPath/help/bigChain.html:Genome_Browser_bigChain_Track_Format	bigChain Track Format The bigChain format describes a pairwise alignment that allow gaps in both sequences simultaneously, just as chain files do; however, bigChain files are compressed and indexed as bigBeds. Chain files are converted to bigChain files using the program bedToBigBed, run with the -as option to pull in a special autoSql (.as) file that defines the fields of the bigChain. The bigChain files are in an indexed binary format. The main advantage of this format is that only those portions of the file needed to display a particular region are transferred to the Genome Browser server. Because of this, bigChain files have considerably faster display performance than regular chain files when working with large data sets. The bigChain file remains on your local web-accessible server (http, https or ftp), not on the UCSC server, and only the portion needed for the currently displayed chromosomal position is locally cached as a "sparse file". If you do not have access to a web-accessible server and need hosting space for your bigChain files, please see the Hosting section of the Track Hub Help documentation. bigChain format definition The following autoSql definition is used to specify bigChain pairwise alignment files. This definition, contained in the file bigChain.as, will be pulled in when the bedToBigBed utility is run with the -as=bigChain.as option. table bigChain "bigChain pairwise alignment" ( string chrom; "Reference sequence chromosome or scaffold" uint chromStart; "Start position in chromosome" uint chromEnd; "End position in chromosome" string name; "Name or ID of item, ideally both human readable and unique" uint score; "Score (0-1000)" char[1] strand; "+ or - for strand" uint tSize; "size of target sequence" string qName; "name of query sequence" uint qSize; "size of query sequence" uint qStart; "start of alignment on query sequence" uint qEnd; "end of alignment on query sequence" double chainScore; "score from chain" ) Note that the bedToBigBed utility uses a substantial amount of memory: approximately 25% more RAM than the uncompressed BED input file. Creating a bigChain track To create a bigChain track, follow these steps: Step 1. If you already have a chain file you would like to convert to a bigChain, skip to Step 3. Otherwise download this example chain file for the human GRCh38 (hg38) assembly. Step 2. Download these autoSql files needed by bedToBigBed: bigChain.as and bigLink.as. Step 3. Download the bedToBigBed and chainToBigChain programs from the UCSC binary utilities directory. Step 4. Use the fetchChromSizes script from the same directory to create a chrom.sizes file for the UCSC database with which you are working (e.g., hg38). Alternatively, you can download the chrom.sizes file for any assembly hosted at UCSC from our downloads page (click on "Full data set" for any assembly). For example, the hg38.chrom.sizes file for the hg38 database is located at http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.chrom.sizes. Here are wget commands to obtain these above files. wget https://genome.ucsc.edu/goldenPath/help/examples/bigChain.as wget https://genome.ucsc.edu/goldenPath/help/examples/bigLink.as wget http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.chrom.sizes wget https://genome.ucsc.edu/goldenPath/help/examples/chr22_KI270731v1_random.hg38.mm10.rbest.chain Step 5. Use the chainToBigChain utility to generate the pre-bigChain and pre-bigLink files: chainToBigChain chr22_KI270731v1_random.hg38.mm10.rbest.chain bigChain.pre bigChain.link.pre Step 6. Create the bigChain and bigLink files using the bedToBigBed utility: bedToBigBed -type=bed6+6 -as=bigChain.as -tab bigChain.pre hg38.chrom.sizes bigChain.bb bedToBigBed -type=bed4+1 -as=bigLink.as -tab bigChain.link.pre hg38.chrom.sizes bigChain.link.bb Step 7. Move the newly created bigChain (bigChain.bb) and bigLink (bigChain.link.bb) files to a web-accessible http, https or ftp location. Step 8. Construct a custom track using a single track line. Note that any of the track attributes listed here are applicable to tracks of type bigBed. The most basic version of the track line will look something like this: track type=bigChain name="My Big Chain" bigDataUrl=http://myorg.edu/mylab/bigChain.bb linkDataUrl=http://myorg.edu/mylab/bigChain.link.bb Step 9. Paste the custom track line into the text box on the custom track management page. The bedToBigBed program can be run with several additional options. For a full list of the available options, type bedToBigBed (with no arguments) on the command line to display the usage message. Examples Example #1 In this example, you will create a bigChain custom track using an existing bigChain file, bigChain.bb, located on the UCSC Genome Browser http server. This file contains data for the hg38 assembly. To create a custom track using this bigChain file: 1. Construct a track line that references the file: track type=bigChain name="bigChain Example One" description="A bigChain file" bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigChain.bb linkDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigChain.link.bb 2. Paste the track line into the custom track management page for the human assembly hg38 (Dec. 2013). 3. Click the "submit" button. Custom tracks can also be loaded via one URL line. This link loads the same bigChain.bb track and sets additional display parameters in the URL: http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&position=chr22_KI270731v1_random &hgct_customText=track%20type=bigChain%20name=Example %20bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigChain.bb %20linkDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigChain.link.bb%20visibility=pack After this example bigChain is loaded in the Genome Browser, click into a chain on the browser's track display. Note that the details page displays information about the individual chains, similar to that which is available for a standard chain track. Example #2 In this example, you will create your own bigChain file from an existing chain input file. 1. Save this chain file to your computer (Step 1 in Creating a bigChain track, above). 2. Save the autoSql files bigChain.as and bigLink.as to your computer (Step 2, above). 3. Download the bedToBigBed and chainToBigChain utilities (Step 3, above). 4. Save the hg38.chrom.sizes text file to your computer. This file contains the chrom.sizes for the human hg38 assembly (Step 4, above). 5. Run the utilities in Steps 5-7, above, to create the bigChain and bigLink output files. 6. Place the newly created bigChain (bigChain.bb) and and bigLink (bigChain.link.bb) files on a web-accessible server (Step 8). 7. Construct a track line that points to the bigChain file (Step 9, above). 8. Create the custom track on the human assembly hg38 (Dec. 2013), and view it in the Genome Browser (Step 10, above). Sharing your data with others If you would like to share your bigChain data track with a colleague, learn how to create a URL by looking at Example 6 on this page. Extracting data from the bigChain format Because the bigChain files are an extension of bigBed files, which are indexed binary files, it can be difficult to extract data from them. UCSC has developed the following programs to assist in working with bigBed formats, available from the binary utilities directory. - bigBedToBed — converts a bigBed file to ASCII BED format. - bigBedSummary — extracts summary information from a bigBed file. - bigBedInfo — prints out information about a bigBed file. As with all UCSC Genome Browser programs, simply type the program name (with no parameters) at the command line to view the usage statement. Troubleshooting If you encounter an error when you run the bedToBigBed program, check your input file for data coordinates that extend past the the end of the chromosome. If these are present, run the bedClip program (available here) to remove the problematic row(s) in your input file before running the bedToBigBed program. 
/goldenPath/help/gbib.html:GBiB	Genome Browser in a Box User's Guide Contents What is Genome Browser in a Box (GBiB)? Getting Started: Setting up Genome Browser in a Box Using Genome Browser in a Box Improving speed and performance Updating Genome Browser in a Box Viewing your own data Customizing your GBiB Sharing Genome Browser in a Box with others User accounts and sessions Troubleshooting common problems Genome Browser in a Box commands Licensing information ------------------------------------------------------------------------ Other resources: - UCSC blog post about GBiB - GBiB introductory video - GBiB introductory video What is Genome Browser in a Box? Genome Browser in a Box (GBiB) is a "virtual machine" of the entire UCSC Genome Browser website that is designed to run on most PCs (Windows, Mac OSX or Linux). GBiB allows you to access much of the UCSC Genome Browser's functionality from the comfort of your own computer. It is particularly directed at individuals who want to use the Genome Browser toolset to view protected data. If it is not human sequencing reads, it usually does not fall under medical data privacy rules, and you should most likely not use GBiB but rather assembly hubs or track hubs. See our mirror page for adiscussion of the advantages/disadvantages of the different methods to customize your genome data display. Differences between GBiB and the Genome Browser While GBiB and the Genome Browser are similar in many ways, there are key differences. In particular, GBiB makes it much easier to visualize sensitive or protected data. Prior to the introduction of GBiB, it was necessary to upload your data to the UCSC Genome Browser website or place the data files on a publicly accessible web server and supply the URL to UCSC in order to view your own data with the Genome Browser. GBiB removes these requirements: none of your data must be uploaded to the UCSC servers, allowing you to use the Genome Browser on personal datasets in situations where it's infeasible to load the data onto a public web server. Rather than installing the entire UCSC genome annotation database (several terabytes of data), GBiB instead depends upon remote connections to various UCSC servers for much of its functionality and data. It connects to the UCSC download server to obtain genomic sequences, liftOver files, and many of the other large data files, and connects to one of UCSC's public MariaDB servers to download data displayed by the various annotation tracks. A few Genome Browser tracks are unavailable on the UCSC public MariaDB servers due to agreements with the data distributors (DECIPHER and LOVD Variants), and thus are unavailable for use with GBiB. The majority of protected data use in the research community currently focuses on the human genomes, primarily the hg19 (GRCh37) assembly and with a growing body of annotation on the newer hg38 (GRCh38) assembly. As a result, GBiB is currently optimized for use with the hg19 assembly. Many other recent genome assemblies can also be viewed, but access may be slower than for optimized assemblies. Access speed may also be impacted by your connection distance from the UCSC server. To improve performance in these situations, GBiB includes a simple tool that allows you to download ("mirror") selected genome annotation tracks to your machine. You can find more information about this tool in the Improving Speed and Performance section. For more background on GBiB see: Haeussler M, Raney BJ, Hinrichs AS, Clawson H, Zweig AS, Karolchik D, Casper J, Speir ML, Haussler D, Kent WJ. Navigating protected genomics data with UCSC Genome Browser in a Box. Bioinformatics. 2015 Mar 1;31(5):764-6. PMID: 25348212; PMC: PMC4341066 Getting Started: Setting up Genome Browser in a Box System requirements GBiB will run on most modern PCs and major operating systems that meet these basic requirements: - The computer must support virtualization (common for most PCs sold after 2010). - A compatible version of the VirtualBox software (version 4.3.6 or higher) must be installed. This software is free to use in many situations. See the VirtualBox wiki for licensing terms and conditions and installation instructions. You must have administrator privileges to install VirtualBox on your computer. - The computer hard disk must have at least 20 GB of free space (more if you plan to mirror many tracks). - Your network firewall must allow outgoing connections to the following servers and ports: - MariaDB connections, used to load tracks not local to your computer: - US server: Port 3306 on genome-mysql.soe.ucsc.edu (128.114.119.174) - European server: Port 3306 on genome-euro-mysql.soe.ucsc.edu (129.70.40.120) - Rsync, used to download track data: - US server: TCP port 873 on hgdownload.soe.ucsc.edu (128.114.119.163) - European server: TCP port 873 on hgdownload-euro.soe.ucsc.edu (129.70.40.99) - Download HTML descriptions on the fly: - US server: TCP port 80 on hgdownload.soe.ucsc.edu (128.114.119.163) - European server: TCP port 80 on hgdownload-euro.soe.ucsc.edu (129.70.40.99) Installation 1. Confirm that your system meets the above requirements. 2. Download GBiB from the Genome Browser store (see Licensing Information). Due to the large size of the gbib.zip product file, the download time may range from 30 minutes to a few hours depending on your Internet connection speed and distance from UCSC. 3. Extract the contents of the gbib.zip file (three files). If desired, the extracted files can be moved to a different directory on your computer, as long as all three files reside in the same directory and are not renamed. Extraction notes: On OSX, do not use the command line tool "unzip" to extract the files; instead double-click on the gbib.zip file in the Finder window. On Windows, you must use a third-party tool such as 7zip or WinRAR to extract the gbib.zip contents, as the file size exceeds the capabilities of the standard Windows extraction tool. 4. Add GBiB to VirtualBox: - Double-click on the browserbox.vbox file extracted from gbib.zip OR - Start VirtualBox, select Machine >> Add, and open the file browserbox.vbox. [Adding GBiB to VirtualBox] If VirtualBox or the Genome Browser displays an error message during the installation process, consult the Troubleshooting section. Starting and using Genome Browser in a Box Starting GBiB To start using GBiB with VirtualBox: 1. Start VirtualBox. 2. Select "browserbox" on the VirtualBox left-hand menu (see "Getting Started" to add GBiB to VirtualBox). 3. Click the "Start" button in the VirtualBox toolbar. [VirtualBox manager start] This will open a black GBiB terminal window with a Linux command line interface: [GBiB terminal window] In most cases, GBiB will auto-update itself after starting for the first time. This update will often include database tables for RefSeq Genes and other tables that are frequently updated at UCSC. The auto-update may take several minutes to complete and the progress will be shown in the terminal window. 4. Open an Internet browser window to 127.0.0.1:1234. We recommend using this URL instead of http://localhost:1234, because most Internet browsers do not send cookies to "http://localhost", which are required to save your browser configuration between sessions. You may want to bookmark 127.0.0.1:1234 for quick future access. If you have correctly set up GBiB, your Internet browser should display the Genome Browser home page. From this page, you can start using GBiB as you would the public Genome Browser website. Consult the UCSC Genome Browser User's Guide for introductory information on using the Genome Browser tools. Stopping GBiB To shut down the GBiB machine (for example, to change configuration options): 1. Close the GBiB terminal window. 2. Select "Send the shutdown signal". 3. Confirm by clicking "OK". [GBiB power off] Improving speed and performance Under certain circumstances the speed and response time of GBiB may be less than optimal. This section offers suggestions for improving the performance of your GBiB installation. Increasing GBiB RAM As a first measure for performance improvement, try increasing the amount of RAM that GBiB is allowed to use. By default, this limit is set to 1 GB (1024 MB). If your machine has enough RAM installed, you may want to increase the GBiB RAM to 2, 4, or even 8 GB. This will usually improve the system responsiveness. To increase the RAM limit: 1. Shut down the GBiB machine (Stopping GBiB). 2. On the VirtualBox Manager window, click Settings >> System. 3. Move the memory slider to the value of your choice, then click "OK". [GBiB RAM] Using the GBiB mirror tool If your GBiB performance is still slow after increasing the RAM, you may be located too far from UCSC. The load time of default tracks ranges from a few seconds on the west coast of the United States to as much as 7 seconds from Europe. In this situation, you may want to use the GBiB mirror tool to download ("mirror") tracks to your machine, which will greatly increase the access speed for those tracks. To use the mirror tool: 1. Click Tools >> Mirror Tracks in the Genome Browser menu. The first time you open the mirror tool, this page may take a while to load. 2. Select the tracks that you typically use by checking the boxes next to the track names. 3. Click Download. In addition to downloading entire track sets, you can also download individual subtracks. The file size of each track is listed next to the track name. If you are unsure of which tracks to select, we recommend the option Default tracks with conservation tables, but no alignments. When downloading large tracks, keep in mind that you cannot delete these tracks and the related data from GBiB once you have downloaded them. If you find that you've started downloading the wrong track or a track that is too large for your machine, you can cancel the download at any point by clicking Cancel Download Now. Depending on your network bandwidth, the download can take several minutes or up to a few hours over a DSL line. During the download, the file gbib-data.vdi will grow in size, and you will not be able to use GBiB. Once the download is complete, the default tracks should load in less than three seconds for a typical genomic position. If you are in your GBiB on the command-line you can use a direct rsync command for files of interest. For example, if you knew you wanted all the GENCODE tracks on hg19 you could run either of the two rsync commands for the North American or European hgdownload servers: sudo rsync hgdownload.soe.ucsc.edu::mysql/hg19/wgEncodeGencode* /data/mysql/hg19/. sudo rsync hgdownload-euro.soe.ucsc.edu::mysql/hg19/wgEncodeGencode* /data/mysql/hg19/. The above commands will rsync all of the files at the UCSC hgdownload server in the hg19 assembly that start with wgEncodeGencode to your GBiB into the hg19 directory. There are some supporting files in a hgFixed directory, such as for the publication tracks, that could be mirrored with such commands. Here is another example for hg38 where the following commands would download all the supporting encRegTfbs and factorbook tables for the Transcription Factor ChIP-seq Clusters track: sudo rsync hgdownload.soe.ucsc.edu::mysql/hg38/encRegTfbs* /data/mysql/hg38/. sudo rsync hgdownload.soe.ucsc.edu::mysql/hg38/factor* /data/mysql/hg38/. You can also download gbdb files in this manner. sudo rsync hgdownload.soe.ucsc.edu::gbdb/hg19/multiz100way/phyloP100way.wib /data/gbdb/hg19/multiz100way/. sudo rsync hgdownload-euro.soe.ucsc.edu::gbdb/hg19/multiz100way/phyloP100way.wib /data/gbdb/hg19/multiz100way/. The above command would copy the phyloP100way track to display in the GBiB from a local file. Offline mode GBiB has an offline mode that is particularly useful when you want to ensure that GBiB no longer connects to the Internet once the initial download and setup are complete (for instance, to comply with corporate IT policy). Before going offline, first mirror all the tracks that you will want to access. Then, in the GBiB terminal window type the command: gbibOffline. This command will remove GBiB's network access to the UCSC MariaDB server and download servers. Once GBiB is in offline mode, the Genome Browser will display an error message if you attempt to access a data file not located on your local disk; therefore, we do not recommend this option for general use. To reactivate Internet access, click on the GBiB terminal window and type the command: gbibOnline. Updating Genome Browser in a Box The software that supports the UCSC Genome Browser is updated every three weeks. These updates include new features and bug fixes for existing features. The track data, on the other hand, are not updated on a regular basis. New tracks and updates to existing tracks are released as they pass UCSC's quality assurance process. The only exceptions to this are GenBank-based tracks, including RefSeq Genes, GenBank mRNAs, and others, which are updated weekly through an automatic process. Automatic updates By default, GBiB is configured to automatically update its files, including software and tracks that you have mirrored. The updates will not affect your custom tracks, user accounts or sessions. We recommend that you leave the auto-update process turned on if Internet connection speed is not an issue, to ensure that you receive all the latest software features and bug fixes, as well as updates to your mirrored tracks. If you are using a DSL line, we recommend turning off automatic updates. Over a slow internet connection, the GBiB update may take several hours to complete, during which time the software will be unusable. To turn off the auto-update process, start GBiB, then type the command gbibAutoUpdateOff in the GBiB terminal window. The auto-update process can be reactivated by typing the command gbibAutoUpdateOn. Manual updates As an alternative to the auto-update process, you can manually update GBiB. To do so, start GBiB and type the command updateBrowser in the GBiB terminal window. This will run the script that updates the GBiB software and any annotation tracks that you have mirrored. A manual update is sometimes an effective solution when GBiB is functioning incorrectly or stops working (see the Troubleshooting section). Viewing your own data In addition to providing access to the standard set of Genome Browser annotation tracks generated by UCSC, GBiB allows you to upload your own data in the form of custom annotation tracks. These tracks can be viewed in the Genome Browser alongside the native UCSC tracks. Uploading your custom annotation tracks to GBiB is similar in many ways to the process used for uploading custom tracks on the public UCSC Genome Browser website. Custom tracks containing smaller data sets can be uploaded to GBiB through the Add Custom Tracks page. For more information on generating and uploading custom tracks, see the Custom Track help page. One big difference, however, is that GBiB has a built-in web server that can communicate directly with your computer. This eliminates the requirement that local big data files be hosted on a separate publicly accessible web server. As a result, your data remains private to your own computer, and will not be available to others unless you grant them access (see Loading local big data tracks and track hubs). Loading local big data tracks and track hubs For improved display performance in the Genome Browser, big data sets are typically stored in a compressed, indexed binary file format such as bigBed, bigWig, BAM, or VCF that contains the data at several resolutions. Unlike custom tracks, in which the entire data set is loaded at once, big data files transmit only the data for the region currently displayed. In order to load and display data in one of these formats, the public Genome Browser website requires big data files to be placed on a publicly accessible web server. However, because GBiB acts as its own web server, your computer can share local big data files directly with GBiB for easy uploading as a custom track. In addition to custom tracks, you can use track hubs and assembly hubs to easily view your data in GBiB. Track hubs are web-accessible directories of genomic data that offer a broader set of configuration and integration options than custom annotation tracks. Assembly hubs are an extension of track hubs and allow you to specify a file containing your novel genomic sequence, in addition to custom annotation data. One strong advantage of track and assembly hubs are that they persist until you delete them, in contrast to custom tracks outside of saved sessions that will automatically expire and be removed from the server after a few days. As with big data tracks, the GBiB built-in web server circumvents the requirement that track and assembly hubs be uploaded to a public accessible server prior to viewing. For more information see Starting a Blat enabled Assembly Hub on GBiB. Loading local files, such as hubs or big data files, requires that GBiB has access the to local folder containing them. To allow GBiB to access one or more of your local folders, follow these steps: Step 1. Shut down the GBiB virtual machine (Stopping GBiB) Step 2. Allow VirtualBox access to one or more directories on your hard disk 1. Click on the "browserbox" entry in the VirtualBox Manager window, then click Settings. 2. Click on Shared Folders. 3. Click on the small "+" icon. 4. Select a directory on your disk under Folder Path / Other. 5. Select the checkbox to give "Read-only" access and make sure the checkbox for "Auto-mount" is selected. 6. Confirm by clicking "OK". 7. Repeat these steps with other folders, as needed. 8. When you are finished, restart GBiB by clicking the "Start" button again. [GBiB shared] To check if your folders are shared, type this address into your web browser: http://127.0.0.1:1234/folders. It should show all shared folders. To obtain the bigDataUrl of any of the files in your shared folders, right-click on any file and select "Copy link address". You can now paste this URL into the Add Custom Tracks page. Or, if you are uploading a track or assembly hub, right-click on your "hub.txt" file and select "Copy link address". You can now paste this URL into the box on the "Connected Hubs" tab of the Track Data Hubs page. Example: Loading a local BAM custom track Here is an example of a custom track in a shared test/ folder that loads a locally hosted BAM file when pasted on the custom tracks page (select My Data >> Custom Tracks in the Genome Browser menu): track type=bam name=BamExample bigDataUrl=http://127.0.0.1:1234/folders/test/bamExample.bam To customize this URL for your own use, replace the URL with a pasted URL to files from your own machine discoverable under the My Data >> GBiB Shared Data Folder in the Genome Browser menu. Since the GBiB is configured so that the VirtualBox shared folder path can be placed directly into the custom track page (via a line udc.localDir=/folders in the hg.conf file) and there is software that knows files ending in .bam should be loaded as type=bam, the link can be replaced with just a path to the file. For example, pasting the following on the custom tracks page would also work: /folders/test/bamExample.bam Connecting to GBiB with ssh The GBiB terminal is a normal Linux command line interface. For easier use of the command line, you can connect to the GBiB machine from your computer with ssh. This may offer better speed and more functionality, such as support for copy/paste. To connect to GBiB with ssh, open a terminal on your computer and type: ssh browser@localhost -p 1235. You will be prompted for a password when attempting to access GBiB from your computer's command line. The password is "browser". Alternatively you can use sudo for root access, which does not require a password. Because stock Windows computers do not have ssh installed, Windows users will have to use the GBiB terminal or install a third-party ssh client for Windows, such as the free software Putty. Data and track conversion tools By default, GBiB includes a few of the commonly used UCSC file manipulation tools, such as bedToBigBed, wigToBigWig, samtools and tabix. These tools can be used to convert and manipulate your basic files into formats that can be uploaded to GBiB as custom tracks. If you need additional Genome Browser tools, type the following command into the GBiB terminal window: gbibAddTools. This command downloads and installs the full suite of command line tools provided by UCSC. Many of these extra tools can be used to extract data and other useful information from your files, or to convert them between various file types. A complete listing and description for all of these tools can be found on UCSC's download server. You can use these tools to convert and extract data from your shared files with the standard "Read-only" settings. However, if you would like to to modify files you've shared with your GBiB, you will have to ensure that the "Read-only" access for VirtualBox is turned off. To do so, follow the directions in Step 2 of the Loading local big data tracks section, but deselect the checkbox next to "Read-only". Please note that the gbibAddTools command requires sudo permissions. Example: Indexing a local BAM file A BAM file must be indexed before it can be loaded into the browser. For example, to index a BAM file in a shared folder "Documents" on your hard disk, type: cd /folders/Documents samtools index my.sorted.bam Example: Using a format conversion tool BED files must be converted to bigBed format to be loaded into the browser. For example, to convert a .bed file in a shared folder "Documents" on your hard disk to a .bigBed format file, type: cd /folders/Documents fetchChromSizes hg19 > hg19.sizes bedToBigBed bedExample.txt hg19.sizes myBigBed.bb Example: Loading a GEO File Some files at external locations, like GEO, are already in binary indexed formats that can be loaded in the browser over the Internet. However, if the server providing these files does not accept byte-range requests, they cannot be transmitted over the Internet to view in the browser. For GEO files, try the ftp location first. If you find the files of interest are giving byte-range request errors, then one option is to download the files and locally load them from your own laptop. With GBiB you can download these files to a local shared folder and then browse them. In the following example the URL of the bigWig (.bw) file was obtained for the wget by right clicking the http link for this GSM1186795 example, that only displays data on chrM and chrX. ssh browser@localhost -p 1235 (to enter GBiB from your computer terminal, password: browser) cd /folders sudo wget "https://www.ncbi.nlm.nih.gov/geo/download/?acc=GSM1186795&format=file&file=GSM1186795%5FDZ012%5FTRA1%5FN2L3%5Ftr27%5Fce10%5FBTm1x200n%2Ebw" -O GSM1186795.bw The file can then be found in the GBiB's http://127.0.0.1:1234/folders/ location, or loaded with a direct link to the files location: http://127.0.0.1:1234/cgi-bin/hgTracks?db=hg19&hgt.customText=http://127.0.0.1:1234/folders/GSM1186795.bw&position=chrM (NOTE: In this above example, the ftp location now has byte-range requests supported so the GEO ftp link can be loaded on the browser). Customizing your GBiB Allowing local-only assemblies By modifying the MySQL table hgcentral.dbDb one can add a genome directly to the UCSC Genome Browser, as documented in our manual mirror instructions. We discourage this, as assembly hubs, especially in combination with a "default cart" (see below) are much easier to setup. However, if you have an existing local genome browser installation with genomes that only exist only there, in order to combine its "remote access" mode with a local-only genome, the local genome has to be declared in hg.conf, or for GBIB, rather /usr/local/apache/cgi-bin/hg.conf.local. The statement is slow-db.excludeDbs=assemblyName1,assemblyName2 For example, if you have two local-only assemblies defined in dbDb with the names homSap1 and homSap2, the statement slow-db.excludeDbs=homSap1,homSap2 will instruct the browser to never try to connect to the UCSC Public MySQL server for these two assemblies, avoiding error messages and significantly speeding up the display. Changing the default Genome Browser options on a GBiB GBiB supports changes to some default settings such as assembly, attached hubs, fonts, text size, etc. Changing these defaults means that any time a new user goes to the mirror, or whenver all user settings are reset, the custom options will be enabled. This can be useful in cases when it is desirable to have hubs attached by default, to change the default assembly or visibility display, etc. The first step is to create a Session containing all desired display options, hubs, etc. Then a file will be created from within the GBiB that contains the command to create a new MySQL table: defaultCart. See the Connect with ssh section of this page for help logging into your GBiB. Contents of new file defaultCart.sql: #The default cart CREATE TABLE defaultCart ( contents longblob not null # cart contents ); The table can then be loaded: mysql hgcentral < defaultCart.sql Finally, the contents of the desired session can be entered into the new table. mysql hgcentral -Ne "insert into defaultCart select contents from namedSessionDb where sessionName='nameOfSession' and userName='nameOfUser'" Where nameOfSession is the name of the created session and nameOfUser is the name of the user the session was created under. Keep in mind that only the top entry in that table will be used to pull default variables, so to change the new defaults (or to remove the new dedaults) you will first want to delete the table contents: mysql hgcentral -Ne "delete from defaultCart" Sharing Genome Browser in a Box with others By default, GBiB can be accessed only from the machine on which it is installed. This is done to prevent others from accessing your data. You can, however, make your GBiB instance available for use by others. To open up external access to GBiB: 1. Shut down the GBiB machine (Stopping GBiB). 2. Select the browserbox machine in the VirtualBox left-hand menubar. 3. Select Machine >> Settings in the VirtualBox menu to display the Settings window. 4. Go to Network >> Adapter 1 >> Advanced >> Port Forwarding. 5. Remove the address "127.0.0.1" from "Rule 1" by deleting it with the backspace key. 6. Click "OK". Note: In addition to enabling port forwarding for VirtualBox, you may need to enable the port forwarding functionality on your PC's firewall to allow others to access your GBiB. You will have to search online for instructions on how to enable this functionality for your PC's firewall. Once you have opened external access to GBiB, your colleagues can access and use your GBiB instance by typing your IP address into their own Internet browser, followed by the :1234 port. Keep in mind that once you have opened up GBiB for remote access, anyone who knows your IP address will be able to access your instance of GBiB and the files that you have shared with it. To control access by others to configure track mirroring in your shared GBiB, you can use the commands gbibMirrorOff and gbibMirrorOn to disable or enable the "Mirror Tracks" function in the menu. User accounts and sessions The Session tool allows you to take a snapshot of your browser configured with specific track combinations, including custom tracks. A browser session can be saved for future use or shared with others who use your GBiB instance. To use the Session tool, you must first create a user account. User accounts and sessions on GBiB are separate from those maintained on the UCSC Genome Browser public website. Because of this, user names and sessions that you create on GBiB cannot be used with the main UCSC Genome Browser website, and vice versa. More information on creating a user account and creating, saving, and sharing sessions can be found in the Sessions User's Guide. Username recovery on accounts is not supported at this time; however, you can recover a lost password. The system for recovering lost passwords on GBiB is much different from that on the Genome Browser and requires access to the command line. To recover a lost password: 1. Navigate to the account login page. 2. Click on the "Can't access your account?" link on the login page. 3. Select the "I forgot my password. Send me a new one." option and enter your username. 4. An email message will be sent to the Alpine email client included with VirtualBox. 5. To access the email client, click on the GBiB terminal window. 6. In this window, type mail and press "enter", which will bring up the Alpine email client. 7. Select MESSAGE INDEX from the menu and press enter. 8. Select the message with "New temporary password..." in the subject line. 9. Log in using your username and this temporary password. 10. After logging in, you will be prompted to create a new password. 11. Once you are finished, exit the Alpine email client by pressing "Q" and then "Y". Please be aware that anyone with access to your username and the command line interface of your GBiB can change your password. Troubleshooting common problems This section addresses some common errors and problems that you may encounter while setting up and installing GBiB. It is not intended as a comprehensive list. If you experience a problem not listed below, please email the UCSC Genome Browser public support mailing list at genome@soe.ucsc.edu. Note that messages sent to this address are publicly accessible. VirtualBox Error: "VT-x/AMD-V hardware acceleration has been enabled, but is not operational. Your 64-bit guest will fail to detect a 64-bit CPU and will not be able to boot." Solution 1: Some older entry-level laptops from around 2009-2011 (e.g. Toshiba Satellite U500) were sold with CPUs that do not support virtualization. These laptops cannot run GBiB. The same applies to low-cost laptops ("netbooks") with Intel Atom processors. Solution 2: On some hardware, virtualization is supported but deactivated in the BIOS. Here is one example of how virtualization support is activated on some hardware. Note that your BIOS virtualization options may differ from those described here. 1. Reboot the computer and press F12 during boot to show the BIOS menu. 2. Go to BIOS Setup >> Virtualization Support >> Virtualization and check "Enable Intel Virtualization Technology". On some Dell systems, you may need to enable additional virtualization options under the BIOS Setup. Go to Virtualization Support >> Virtualization for Direct I/O and check "Enable Virtualization for Direct I/O". 3. Exit and save, then restart the computer. VirtualBox Error: "Failed to open virtual machine located in... Trying to open a VM config ... which has the same UUID as an existing virtual machine." Solution: This error occurs if GBiB has been previously downloaded and installed. To resolve this problem: 1. Start VirtualBox. 2. Select the currently installed version of browserbox from the left-hand column. 3. Select Machine >> Remove (Ctrl+R or ⌘+R) in the VirtualBox menu. When asked, choose "Remove only" to retain the old browserbox version on your disk. 4. Double-click the newly downloaded browserbox.vbox file or add it with the Machine >> Add menu option. VirtualBox Error: "Failed to open virtual machine located in... Cannot register the hard disk ... because a hard disk ... already exists." Solution: This error occurs if GBiB has been previously downloaded and installed. To resolve this problem: 1. Start VirtualBox. 2. Select File >> Virtual Media Manager (Ctrl+D or ⌘+D) in the VirtualBox menubar. 3. Select gbib-data.vdi and click "Remove". 4. Double-click the newly downloaded browserbox.vbox file or add it with Machine >> Add menu option. Genome Browser Error: "Couldn't connect to database hg19 on genome-mysql.soe.ucsc.edu as genomep." Solution 1: This indicates that the virtual machine could not connect to the UCSC MariaDB server. This error can be caused by a change of the IP address (e.g. on a wifi connection) that has not yet been picked up by the virtual machine. In this situation, you can restart the box or run the command sudo ifup --force eth0 to reset the network connection. Solution 2: Alternatively, this error may be generated when the firewall does not allow outgoing TCP data on port 3306/MySQL. In this case, contact your institution's IT support staff to inquire about ways to open this port. Genome Browser Error: "Couldn't connect to database hgcentral on localhost as root. Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (13)" Solution: This error can be caused when the virtual machine is downloading data such as when using the mirror tool to install local copies of tracks to increase speed and performance. Once the download finishes, the message will no longer appear. If one cancels a download in progress, the error can persist, in which case restart your virtual machine and the message should be removed. Problem: Error when using BLAT. When you use BLAT on your GBiB machine, it attempts to open outgoing TCP connections to the BLAT servers running at UCSC. Each assembly has two ports that need to be open from your GBiB machine to the UCSC BLAT servers. You will likely need to contact the system administrators at your institution and ask them to open outgoing TCP connections to a list of UCSC hostnames and ports. The commands below will allow you to find out the names of these hosts and ports. Solution: To find the hostname-port combination to open from your GBiB machine, use this SQL query: mysql hgcentral -e 'select * from blatServers where db="YOURDB"' For example, if you want to enable BLAT for hg19 and hg38, you can issue this command: mysql hgcentral -e 'select * from blatServers where db="hg38" or db="hg19"' +------+---------------------+-------+---------+--------+ | db | host | port | isTrans | canPcr | +------+---------------------+-------+---------+--------+ | hg19 | blat4a.soe.ucsc.edu | 17779 | 0 | 1 | | hg19 | blat4a.soe.ucsc.edu | 17778 | 1 | 0 | | hg38 | blat4c.soe.ucsc.edu | 17781 | 0 | 1 | | hg38 | blat4c.soe.ucsc.edu | 17780 | 1 | 0 | +------+---------------------+-------+---------+--------+ Problem: GBiB is functioning incorrectly or stops working. Solution 1: If you have previously turned off automatic updates, your software or data may need updating. Try manually updating GBiB. This update process may take several hours over a slow Internet connection. Solution 2: Re-download the gbib.zip file and extract only the file gbib-root.vdi. Place this file in the same directory with the files extracted from your original GBiB installation. Do not extract and overwrite gbib-data.vdi -- it contains your personal track and session settings and mirrored tracks. Problem: I need proxy support for my files to load. Solution: Proxy servers may be required by some installations to get through a firewall. You can add the settings httpProxy, httpsProxy and ftpProxy to hg.conf.local at /usr/local/apache/cgi-bin/hg.conf.local: httpProxy=http://someProxyServer:3128 httpsProxy=http://someProxyServer:3128 ftpProxy=ftp://127.0.0.1:2121 If the proxy server requires BASIC authentication, then the line in hg.conf.local should look like this: httpProxy=http://user:password@someProxyServer:3128 httpsProxy=http://user:password@someProxyServer:3128 If there are domains or domain-suffices that should not be proxied, use noProxy. noProxy=ucsc.edu,mit.edu,localhost,127.0.0.1 The file /usr/local/apache/cgi-bin/hg.conf should already include a line like include hg.conf.local to incorporate the changes in hg.conf.local. Problem: "No space left on device" error when running gbibAddTools. In older GBiB's, there is an error in the gbibAddTools command. To fix this command, edit the /home/browser/.bashrc file and change the following line: alias gbibAddTools='mkdir ~/bin -p; rsync -avP hgdownload.soe.ucsc.edu::genome/admin/exe/linux.x86_64/ ~/bin/' to either of the follwing lines: alias gbibAddTools='sudo mkdir -p /data/tools; sudo rsync -avP hgdownload.soe.ucsc.edu::genome/admin/exe/linux.x86_64/ /data/tools/ && ln -s /data/tools ~/bin' alias gbibAddTools='sudo mkdir -p /data/tools; sudo rsync -avP hgdownload-euro.soe.ucsc.edu::genome/admin/exe/linux.x86_64/ /data/tools/ && ln -s /data/tools ~/bin' The updated commands will install the tools to a location with more disk space available using either the North American or European hgdownload servers. Genome Browser in a Box commands In addition to normal Linux commands, GBiB defines some special commands you may use while inside the GBiB terminal's window. These additional commands are documented in the README.txt file on the GBiB terminal's home directory, which can also be accessed via ssh. General commands --------------------- ------------------------------------------------------------------------------------------------ gbibAutoUpdateOff Switch off automatic weekly updates gbibAutoUpdateOn Reactivate automatic weekly updates gbibOffline Switch off remote access to UCSC for tables or files gbibOnline Reactivate remote access to UCSC for tables or files gbibMirrorTracksOff Disable the "Mirror tracks" tool gbibMirrorTracksOn Enable the "Mirror tracks" tool gbibAddTools Download the UCSC genome command line tools into the ~bin directory. Requires sudo permissions --------------------- ------------------------------------------------------------------------------------------------ Advanced commands --------------------- -------------------------------------------------------------------------------------------- gbibCoreUpdate Download the most current update script from UCSC now, this is part of an automatic update gbibFixMysql1 Fix all MariaDB databases, fast version gbibFixMysql2 Fix all MariaDB databases, intensive version gbibResetNetwork Reinit the eth0 network interface, in case VirtualBox dropped the network connection gbibUcscLog Show a real-time log of all SQL queries on the console gbibUcscTablesLog Show the tables that had to be loaded through the internet from UCSC gbibUcscTablesReset Reset the table counters gbibUcscGbdbLog Show the gbdb files that had to be loaded through the internet from UCSC gbibUcscGbdbReset Reset the gbdb counters --------------------- -------------------------------------------------------------------------------------------- Licensing information GBiB is free for non-profit academic research and for personal use. Corporate use requires a license, setup fee and annual payment. To purchase a license or download the GBiB, visit the Genome Browser store. 
/goldenPath/help/bam.html:Genome_Browser_BAM_Track_Format	BAM Track Format BAM is the compressed binary version of the Sequence Alignment/Map (SAM) format, a compact and index-able representation of nucleotide sequence alignments. Many next-generation sequencing and analysis tools work with SAM/BAM. For custom track display, the main advantage of indexed BAM over PSL and other human-readable alignment formats is that only the portions of the files needed to display a particular region are transferred to UCSC. This makes it possible to display alignments from files that are so large that the connection to UCSC would time out when attempting to upload the whole file to UCSC. Both the BAM file and its associated index file remain on your web-accessible server (http, https, or ftp), not on the UCSC server. UCSC temporarily caches the accessed portions of the files to speed up interactive display. If you do not have access to a web-accessible server and need hosting space for your BAM files, please see the Hosting section of the Track Hub Help documentation. The typical workflow for generating a BAM custom track is this: 1. If you haven't done so already, download and build the samtools program. Test your installation by running samtools with no command line arguments; it should print a brief usage message. For help with samtools, please contact the SAM tools mailing list. 2. Align sequences using a tool that outputs SAM directly, or outputs a format that can be converted to SAM. (See list of tools and converters) 3. Convert SAM to BAM using the samtools program: samtools view -S -b -o my.bam my.sam If converting a SAM file that does not have a proper header, the -t or -T option is necessary. For more information about the command, run samtools view with no other arguments. 4. Sort and create an index for the BAM: samtools sort -o my.sorted.bam my.bam samtools index my.sorted.bam The sort command appends .bam to my.sorted, creating a BAM file of alignments ordered by leftmost position on the reference assembly. The index command generates a new file, my.sorted.bam.bai, with which genomic coordinates can quickly be translated into file offsets in my.sorted.bam. 5. Move both the BAM file and index file (my.sorted.bam and my.sorted.bam.bai) to an http, https, or ftp location. Note that the Genome Browser looks for an index file with the same URL as the BAM file with the .bai suffix added. If your hosting site does not use the filename as the URL link, you will have to specifically call the location of this .bam.bai index file with the bigDataIndex keyword. This keyword is relevant in Custom Tracks and Track Hubs. You can read more about bigDataIndex in the TrackDb Database Definition page. 6. If the file URL ends with .bam and the BAM index file URL ends with .bam.bai, you can paste the URL directly into the custom track management page, click submit and view in the Genome Browser. The track name will then be the name of the file. If you want to configure the track name and descriptions or the URLs are not as described, you will need to create a track line, as shown below in step 7. 7. Construct a custom track using a single track line. The most basic version of the "track" line will look something like this: track type=bam name="My BAM" bigDataUrl=http://myorg.edu/mylab/my.sorted.bam Again, in addition to http://myorg.edu/mylab/my.sorted.bam, the associated index file http://myorg.edu/mylab/my.sorted.bam.bai must also be available at the same location. If not, you can specify the URL with the bigDataIndex=http://myorg.edu/mylab/my.sorted.bam.bai 8. Paste the custom track line into the text box in the custom track management page, click submit and view in the Genome Browser. Parameters for BAM custom track definition lines All options are placed in a single line separated by spaces. In the example below, the lines are broken only for readability. If you copy/paste this example, you must remove the line breaks. Click here for a text version that you can paste without editing. track type=bam bigDataUrl=http://... pairEndsByName=. pairSearchRange=N bamColorMode=strand|gray|tag|off bamGrayMode=aliQual|baseQual|unpaired bamColorTag=XX minAliQual=N showNames=on|off name=track_label description=center_label visibility=display_mode priority=priority db=db maxWindowToDraw=N chromosomes=chr1,chr2,... The track type and bigDataUrl are REQUIRED: type=bam bigDataUrl=http://myorg.edu/mylab/my.sorted.bam The remaining settings are OPTIONAL. Some are specific to BAM: pairEndsByName any value # presence indicates paired-end alignments pairSearchRange N # max distance between paired alignments, default 20,000 bases bamColorMode strand|gray|tag|off # coloring method, default is strand bamGrayMode aliQual|baseQual|unpaired # grayscale metric, default is aliQual bamColorTag XX # optional tag for RGB color, default is "YC" minAliQual N # display only items with alignment quality at least N, default 0 showNames on|off # if off, don't display query names, default is on Other optional settings are not specific to BAM, but relevant: name track label # default is "User Track" description center label # default is "User Supplied Track" visibility squish|pack|full|dense|hide # default is hide (will also take numeric values 4|3|2|1|0) bigDataUrl https://your.bam.bai.com # default is the bigDataUrl with .bai added to the end priority N # default is 100 db genome database # e.g. hg18 for Human Mar. 2006 maxWindowToDraw N # don't display track when viewing more than N bases chromosomes chr1,chr2,... # track contains data only on listed reference assembly sequences doWiggle on|off # if on, show data as density graph, default is off The BAM track configuration help page describes the BAM track configuration page options corresponding to pairEndsByName, minAliQual, bamColorMode, bamGrayMode and bamColorTag in more detail. pairSearchRange applies only when pairEndsByName is given. It allows for a tradeoff of display speed vs. completeness of pairing the paired-end alignments. When paired ends are split or separated by large gaps or introns, but one is viewing a small genomic region, it is necessary to search a large number of bases upstream and downstream of the viewed region in order to find mates of the alignments in the viewed region. However, searching a very large region can be slow, especially when the alignments have deep coverage of the genome. To ensure that all properly paired mates will be found, pairSearchRange should be set to the largest genomic size of a mapped pair. However, it can be set to a smaller size if necessary to speed up the display, at the cost of some items being displayed as unpaired when the mate is too far outside the viewed window. Example #1 In this example, you will create a custom track for an indexed BAM file that is already on a public server — alignments of sequence generated by the 1000 Genomes Project. You can paste the URL http://genome.ucsc.edu/goldenPath/help/examples/bamExample.bam directly into the custom track management page for the human assembly hg18 (May 2006), then press the submit button. On the following page, press the chr21 link in the custom track and navigate to position chr21:33,038,946-33,039,092 to see the reads in the new BAM track. Alternatively, you can specify more visualization options by creating a "track" line. The line breaks inserted here for readability must be removed before submitting the track line: track type=bam name="BAM Example One" description="Bam Ex. 1: 1000 Genomes read alignments (individual NA12878)" pairEndsByName=. pairSearchRange=10000 chromosomes=chr21 bamColorMode=gray maxWindowToDraw=200000 db=hg18 visibility=pack bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bamExample.bam Include the following "browser" line to view a small region of chromosome 21 with alignments from the .bam file: browser position chr21:33,038,946-33,039,092 Note if you copy/paste the above example, you must remove the line breaks (or, click here for a text version that you can paste without editing). Paste the "browser" line and "track" line into the custom track management page for the human assembly hg18 (May 2006), then press the "submit" button. On the following page, press the chr21 link in the custom track listing to view the BAM track in the Genome Browser. Example #2 In this example, you will create indexed BAM from an existing SAM file. First, save this SAM file samExample.sam to your machine. Perform steps 1 and 3-7 in the workflow described above, but substituting samExample.sam for my.sam. On the custom track management page, click the "add custom tracks" button if necessary and make sure that the genome is set to Human and the assembly is set to Mar. 2006 (hg18) before pasting the track line and submitting. This track line is a little nicer than the one shown in step 6, but remember to remove the line breaks that have been added to the track line for readability (or, click here for a text version that you can paste without editing): track type=bam name="BAM Example Two" bigDataUrl=http://myorg.edu/mylab/my.sorted.bam description="Bam Ex. 2: Simulated RNA-seq read alignments" visibility=squish db=hg18 chromosomes=chr21 browser position chr21:33,037,317-33,038,137 browser pack mrna Sharing Your Data with Others If you would like to share your BAM data track with a colleague, learn how to create a sharable URL by looking at this page. 
/goldenPath/help/bigNarrowPeak.html:Genome_Browser_bigNarrowPeak_Track_Format	bigNarrowPeak Track Format bigNarrowPeak is a format used to provide called peaks of signal enrichment based on pooled, normalized (interpreted) data. The bigNarrowPeak format stores annotation items that are a single block with a single base peak within that block, much as BED files indexed as bigBeds do. A bigNarrowPeak file is a standard six field bed with four additional fields (BED6+4 format) that contain three doubles with scoring information and the location of the single base peak. It is the binary version of the ENCODE narrowPeak or point-source peak format. The bigNarrowPeak files are created using the program bedToBigBed, run with the -as option to pull in a special autoSql (.as) file that defines the extra fields of the bigNarrowPeak. The bigNarrowPeak files are in an indexed binary format. The main advantage of this format is that only those portions of the file needed to display a particular region are transferred to the Genome Browser server. Because of this, indexed binary files have considerably faster display performance than regular BED format files when working with large data sets. The bigNarrowPeak file remains on your local web-accessible server (http, https or ftp), not on the UCSC server, and only the portion needed for the currently displayed chromosomal position is locally cached as a "sparse file". If you do not have access to a web-accessible server and need hosting space for your bigNarrowPeak files, please see the Hosting section of the Track Hub Help documentation. bigNarrowPeak file definition The following autoSql definition is used to specify bigNarrowPeak files. This definition, contained in the file bigNarrowPeak.as, is pulled in when the bedToBigBed utility is run with the -as=bigNarrowPeak.as option. table bigNarrowPeak "BED6+4 Peaks of signal enrichment based on pooled, normalized (interpreted) data." ( string chrom; "Reference sequence chromosome or scaffold" uint chromStart; "Start position in chromosome" uint chromEnd; "End position in chromosome" string name; "Name given to a region (preferably unique). Use . if no name is assigned" uint score; "Indicates how dark the peak will be displayed in the browser (0-1000) " char[1] strand; "+ or - or . for unknown" float signalValue; "Measurement of average enrichment for the region" float pValue; "Statistical significance of signal value (-log10). Set to -1 if not used." float qValue; "Statistical significance with multiple-test correction applied (FDR -log10). Set to -1 if not used." int peak; "Point-source called for this peak; 0-based offset from chromStart. Set to -1 if no point-source called." ) Click here to view an example of a bigNarrowPeak (bed6+4) input file. Note that the bedToBigBed utility uses a substantial amount of memory: approximately 25% more RAM than the uncompressed BED input file. Creating a bigNarrowPeak track To create a bigNarrowPeak track, follow these steps: Step 1. Create a bigNarrowPeak file. The first six fields of the bigNarrowPeak bed6+4 format are described by the basic BED file format shown here. You can also read about narrowPeak (or point-source peak), the precursor to bigNarrowPeak, here. Your bigNarrowPeak file must also contain the four extra fields described in the autoSql file definition shown above: signalValue, pValue, qValue, peak. Your bigNarrowPeak file must be sorted first on the chrom field, and secondarily on the chromStart field. You can use the UNIX sort command to do this: sort -k1,1 -k2,2n unsorted.bed > input.bed Step 2. Download the bedToBigBed program from the binary utilities directory. Step 3. Use the fetchChromSizes script from the same directory to create a chrom.sizes file for the UCSC database with which you are working (e.g., hg38). Alternatively, you can download the chrom.sizes file for any assembly hosted at UCSC from our downloads page (click on "Full data set" for any assembly). For example, the hg38.chrom.sizes file for the hg38 database is located at http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.chrom.sizes. Step 4. Create the bigNarrowPeak file from your sorted input file using the bedToBigBed utility: bedToBigBed -as=bigNarrowPeak.as -type=bed6+4 bigNarrowPeak.txt chrom.sizes myBigNarrowPeak.bb Step 5. Move the newly created bigNarrowPeak file (myBigNarrowPeak.bb) to a web-accessible http, https, or ftp location. Step 6. Construct a custom track using a single track line. Note that any of the track attributes listed here are applicable to tracks of type bigNarrowPeak. The basic version of the track line will look something like this: track type=bigNarrowPeak name="My Big NarrowPeak" description="A Set of Peaks from DNase Experiments" bigDataUrl=http://myorg.edu/mylab/myBigNarrowPeak.bb Step 7. Paste this custom track line into the text box on the custom track management page. The bedToBigBed program can be run with several additional options. For a full list of the available options, type bedToBigBed (with no arguments) on the command line to display the usage message. If you do not have access to a web-accessible server and need hosting space for your bigNarrowPeak files, please see the Hosting section of the Track Hub Help documentation. Examples Example #1 In this example, you will create a bigNarrowPeak custom track using a bigNarrowPeak file, bigNarrowPeak.bb, located on the UCSC Genome Browser http server. This file contains data for the hg38 assembly. To create a custom track using this bigNarrowPeak file: 1. Construct a track line that references the file: track type=bigNarrowPeak name="bigNarrowPeak Example One" description="A bigNarrowPeak file" bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigNarrowPeak.bb 2. Paste the track line into the custom track management page for the human assembly hg38 (Dec. 2013). 3. Click the "submit" button. Custom tracks can also be loaded via one URL line. The link below loads the same bigNarrowPeak track and sets additional parameters in the URL: http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&hgct_customText=track%20type=bigNarrowPeak %20name=Example%20bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigNarrowPeak.bb After this example bigNarrowPeak track is loaded in the Genome Browser, click on a peak in the browser's track display to view the details page for that peak. Example #2 In this example, you will create your own bigNarrowPeak file from an existing bigNarrowPeak input file. 1. Save the example bed6+4 input file, bigNarrowPeak.txt, to your computer (Step 1 in Creating a bigNarrowPeak track, above). 2. Download the bedToBigBed utility (Step 2, above). 3. Save the hg38.chrom.sizes text file to your computer. This file contains the chrom.sizes for the human hg38 assembly (Step 3, above). 4. Save the autoSql file bigNarrowPeak.as to your computer. 5. Run the bedToBigBed utility to create the bigNarrowPeak output file (step 4, above): bedToBigBed -type=bed6+4 -tab -as=bigNarrowPeak.as bigNarrowPeak.txt hg38.chrom.sizes bigNarrowPeak.bb 6. Place the newly created bigNarrowPeak file (bigNarrowPeak.bb) on a web-accessible server (Step 5, above). 7. Construct a track line that points to the bigNarrowPeak file (Step 6, above). 8. Create the custom track on the human assembly hg38 (Dec. 2013), and view it in the Genome Browser (step 7, above). Example #3 In this example, you will see the additional filtering options available for the signalValue, pValue, qValue fields in the bigNarrowPeak format. 1. Look to the above constructed track line in example one that references the bigNarrowPeak.bb file. 2. Now look at the below version that adds the following filters signalFilter, pValueFilter, qValueFilter to the track line as well as setting limitations on those filters with additional signalFilterLimits, pValueFilterLimits, qValueFilterLimits settings. track type=bigNarrowPeak name="bigNarrowPeak Example Filter" description="A bigNarrowPeak file with additional Filter Settings" signalFilter=0 signalFilterLimits=0:18241 pValueFilter=2 pValueFilterLimits=0:300 qValueFilter=2 qValueFilterLimits=0:300 bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigNarrowPeak.bb These values are simply added to the earlier track line in pairs, such as pValueFilterLimits=0:300, and can be given a set value for where the filter should start such as pValueFilter=2. 3. Paste the track line from above into the custom track management page for the human assembly hg38 (Dec. 2013). 4. Click the "submit" button. 5. When viewing the track, right-click the track and then select the "Configure bigNarrowPeak Example One" option and you will see you can enter different filtering options. These same options could also be used in a track hub using the bigNarrowPeak format. Here is an example track hub stanza: track exBigNarrowPeakTrack type bigNarrowPeak visibility full signalFilter 0 signalFilterLimits 0:10000 pValueFilter 0 pValueFilterLimits 0:300 qValueFilter 0 qValueFilterLimits 0:300 shortLabel Ex bigNPk longLabel bigNarrowPeak Example bigDataUrl http://genome.ucsc.edu/goldenPath/help/examples/bigNarrowPeak.bb Sharing your data with others If you would like to share your bigNarrowPeak data track with a colleague, learn how to create a URL by looking at Example #6 on this page and the additional URL optional parameters section. Extracting data from bigBed format Because the bigNarrowPeak files are an extension of bigBed files, which are indexed binary files, it can be difficult to extract data from them. UCSC has developed the following programs to assist in working with bigBed formats, available from the binary utilities directory. - bigBedToBed — converts a bigBed file to ASCII BED format. - bigBedSummary — extracts summary information from a bigBed file. - bigBedInfo — prints out information about a bigBed file. Such tools can be used to obtain only features within a given range, for example: bigBedToBed http://hgdownload.soe.ucsc.edu/gbdb/danRer10/transMap/V4/danRer10.refseq.transMapV4.bigPsl -chrom=chr6 -start=0 -end=1000000 stdout </> As with all UCSC Genome Browser programs, type the program name (with no parameters) at the command line to view the usage statement. Troubleshooting If you encounter an error when you run the bedToBigBed program, check your input file for data coordinates that extend past the end of the chromosome. If these are present, run the bedClip program (available here) to remove the problematic row(s) before running the bedToBigBed program. 
/goldenPath/help/trackSearch.html:Genome_Browser_Track_Search	Track Search The Genome Browser's Track Search feature allows users to find and display tracks of interest quickly and easily. Searching The track search feature provides users with two search options, "Search" and "Advanced." If multiple terms are entered, only tracks with all terms will be part of the results. We do not currently support either/or searching (e.g. a OR b). Search tab: "Search" is the default tab of the track search feature. To search for a track, enter search terms into the box and click search. This search looks for the user's terms in the track names, descriptions, groups, and metadata. Metadata is information, such as cell line, experiment type, and treatment, about ENCODE data. Advanced tab: The "Advanced" tab allows users to search for terms in a specific aspect of the track (e.g., the track name, the description, and the track group) and can be especially helpful for finding ENCODE tracks by setting specific criteria in the "ENCODE terms" section of this tab. The ENCODE terms section of the Advanced tab (which is not available in all assemblies) allows users to further refine their search for ENCODE tracks using ENCODE metadata. By default, there are two rows of ENCODE metadata search criteria, but more can be added by clicking the "+" button. To remove an added metadata search row, click the "-" button. Each row is comprised of a drop-down menu followed by either a second drop-down menu or a text field. The first drop-down menu contains the searchable metadata categories available. The type of metadata term selected in the first drop-down determines the type of search available. Some metadata will have a second drop-down containing the possible metadata terms in that category available to search on. Most of these drop downs are multi-select lists, so users can select one or multiple terms. Other metadata types will have a text field for users to enter specific search terms. There are two types of text fields. If the phrase preceding the text field is "contains," then the field is a free text field, with the option of using the percent sign (%) as a wildcard. If the phrase "is among" precedes the text field, then it is possible to enter a comma-separated list to find all tracks with any of the values in the list in that metadata category. This comma-separated list also accepts the percent sign (%) as a wildcard. Because these metadata drop-downs are assembly dependent, the options available may vary from assembly to assembly. Some categories have a link to the far right of the row if there is more information available about the terms. Results After clicking the "Search" button, the first 100 tracks that meet the search criteria are displayed. Check-box/Visibility: Tracks that are currently being displayed in the browser will have their check-box selected. To turn tracks on or off, select or deselect the associated check-box. To turn all tracks listed on the current results page on or off, click the "+" or "-" button at the top of the column. By default, selecting the check-box of a track turns the track to pack as long as pack is a valid view mode of the track. If it isn't, the track is set to full or show. Once the check-box of a track is selected, the visibility can be changed using the drop-down menu in the visibility column. The [] icon designates "container tracks" that can be configured by clicking the icon. Container tracks don't have data themselves; instead, they contain other tracks that do have data. Examples of container tracks are composite tracks, which have a light orange background in the search results, and super-tracks, which have a tan background; tracks that contain data have a yellow background. While tracks within a container track may show up in the search results and can be configured from the search results page, this isn't always the case. Clicking [] opens the container track configuration page, allowing the all the tracks within the container track to be viewed and configured. Please note, since container tracks don't have data of their own, turning one on via track search may result in no additional tracks in the browser. If this occurs, it is because none of the tracks within the container were turned on. Track information: To find out more information about a track, click on the Track Name. The description of the track or the container track will pop-up. Click "x" at the top of the pop-up or the "OK" at the bottom to close it. For tracks that have metadata, mainly ENCODE tracks at this time, there is a "..." link on the right-hand side of the track result row. Clicking this link expands the row to show the track's metadata. Sorting: By default, resulting tracks are listed in order of relevance. However, in the blue header of the results list, the sort order can be changed by selecting one of the other radio buttons, "Alphabetically" or "by Hierarchy" (super-tracks listed first, then composite tracks, and then data tracks). Return to Browser/View in Browser: The "Return to Browser" button indicates that no tracks have been turned on or off. Clicking on the button returns the user to the browser. Once any track is turned on or off using the check-boxes, the button label changes to "View in Browser" (indicating changes have been made), and clicking on the button takes the user to the browser with the new track selections displayed. 
/goldenPath/help/sessions.html:Genome_Browser_Session_Gallery	Session Gallery Session Gallery - DNA/Codon View - Evolution - SNPs and Disease - Expression/Regulation: TFBS/RNA-seq/ChIP-seq - Mailing List Support More Information about Sessions - What is a Session? - What is the Session Gallery? - Don't copy that link -- make a Session! - Custom Track Caveats DNA/codon view Examples of stop codons in the hg19 assembly [] The above session helps illustrate the display of how one gene can have different transcripts with different stop codons, displayed in red in the browser. Note that the three transcripts that terminate near the right side of the screen are in a different frame from the transcript that terminates near the left side. The three potential reading frames through the region can be seen at the top of the graphic. Only one of the three is free of stop codons throughout this range. Premature Stop in the hg18 Reference Assembly [] A premature stop codon was found in the hg18 reference genome and in about half of people of European descent. The UCSC Genes track was forced to skip the codon to indicate a full-length coding region for this gene. The SNP indicated by the G nucleotide in many of the mRNA alignments shows that in many samples from Genbank, the T > G transversion encodes a glutamic acid in the protein, reading through the premature stop predicted by the reference assembly. [] Clicking the top highlighted rs4940595 in the Human Genome Diversity Project SNP Population Allele Frequencies track will display the world frequency image regarding this variant. In the more recent hg38 reference genome this area displays differently. Click into the active session and then into the SNP to read the details. Link to region in hg38 for comparison. Specifically, in hg38 the T in the previous reference assembly has been replaced by a G, giving the reference a glutamic acid, E, and no longer prematurely truncates the predicted transcript. The SNP at this location, rs4940595, now indicates the G > T transversion in the reverse direction from previously, indicating a premature stop at this location for the T allele. Examples of Start Codons in the hg19 Assembly [] The above session helps illustrate the display of how one gene can have different transcripts with different start codons, displayed in green in the browser. Here is an example session displaying a start codon on the reverse strand. By clicking the "reverse" button below the browser one can flip the view. Examples of Split Codons in the hg19 Assembly [] The above session helps illustrate how a codon can be split between splice sites. By clicking the little double-headed arrow on the left of one of the transcripts one can see the other part of the codon. Note that there are different AAs (G or R) depending on the different splicing upstream for different isoforms. [] Using the multi-region exon-only feature available under the top blue menu bar and View selecting "Multi-Region" you can remove introns adjusting padding down to just one base (or none) to more clearly see the split codon above. Examples of Different Codon Numbering in the hg19 Assembly [] The above session helps illustrate the how same region in an exon can have different codon numbering reflecting different isoforms and that it is important to use caution when reading codon numbers in the literature. Here is an additional view of the region zoomed out 300x to display the alternate splicing. Evolution Examples of Codon Wobble in the hg19 Assembly [] The above session helps illustrate how codon wobble in bases diverges through evolution when the amino acid is the same (note the conservation score graph). There is a conservation of function through evolution with V > I > F amino acids and K > R substitutions. Here is an additional example session displaying wobble. Examples of Evolution in the hg19 Assembly [] This session shows alignments of three species (Chimp, Gorilla, and Orangutan) against chr2 in hg19 and how the same sequences are found in two separate chromosomes in the other species. [] Zooming in above to a region of human chr2 you can see heterochromatin around a vestigial centromere which has not evolved away yet. By using the top blue menu bar under View to select "In Other Genomes (Convert)" you can select the Chimp assembly panTro4 and see that these coordinates in hg19 (chr2:126,645,017-138,651,416) coincide with the centromere in the Chimp chr2B: [] SNPs and disease Examples of SNP coloring and ABO variant details in the hg19 Assembly [] The above session helps illustrate how SNPs are color coded regarding whether they represent a nonsense, missense, or frameshift variation. These SNPs are around the ABO gene and represent blood group variation. By clicking into the gene track for one of the isoforms of the ABO gene one can see additional variant details by clicking the "Gene Alleles" link to jump down to the "Common Gene Haplotype Alleles" section and mouseover highlighted variant amino acids to see changes in nucleotide and AA sequences. [] Examples of OMIM allelic variants in the hg19 Assembly [] The top track in the above session displays OMIM allelic variants. By clicking the items one can learn the details and find additional links to the allelic variants in the Online Mendelian Inheritance in Man (OMIM) database. In the above and this zoomed in session you can mouseover OMIM variants to access details such as OMIM Allelic Variant 136350.0021 in the far right to see "PRO366LEU,rs121909641:HYPOGONADOTROPIC HYPOGONADISM 2 WITH ANOSMIA, SUSCEPTIBILITY TO" to learn additional information about each variant. Examples of CAG Repeats in Huntingtin in the hg19 Assembly [] The above session uses the Short Match Track to display matches of "cagcagcagcagcagcag" repeats in a polyglutamine region (Note Qs) of the gene HTT, Huntingtin, linked to Huntington's disease. Examples of Segmental Duplications in the hg19 Assembly [] The above session uses the Segmental Dups to display regions detected as putative genomic duplications within the golden path and shows X-linked colorblindness as a recombination hotspot in the opsin region where genes encode for light absorbing visual pigments. Another example featuring the Segmental Dups track is the 15q11 Prader-Willi region in which same-chromosome duplications lead to gain and loss of copies and a rare genetic disorder when some genes are deleted or unexpressed. Expression/regulation: TFBS/RNA-seq/ChIP-seq Microarray Probset data in the hg19 Assembly [] The above session uses the Microarray Probesets track to display Affymetrix, Agilent, and Illumina probe information. View ENCODE data (2003 - 2012) in the UCSC Genome Browser [] The above session highlights some key integrated track data from the ENCODE project including GENCODE gene sets, Transcription Levels by RNA-seq, H3K27Ac histone mark, DNaseI Hypersensitivity, ChIP-seq Transcription Factor binding sites, and Genome Segmentations. Example search of enhancer information around the gene TNFAIP3 [] This session comes from a question regarding how to search for enhancer RNA for the gene TNFAIP3. The response displays the various ENCODE tracks around the gene TNFAIP3 indicating where DNase activity, Transcription Factor Binding, RNA transcription, and histone modification has been observed. The layered H3K4Me1 and layered H3K27Ac tracks show where modification of histone proteins is suggestive of enhancer and other regulatory activity. Investigating where RNA-seq signals overlap with Transcription Factor Binding upstream of TNFAIP3 transcription would appear to indicate possible eRNA locations and the segmentation track displays possible Strong and Weak Enhancers as calculated by the ChromHMM and Segway programs. Example of how DNaseI Hypersensitivity Clusters track represents multiple experiments [] This session comes from a question about how the DNaseI Hypersensitivity Clusters track is generated. The response explains that the clusters track represents the combining and filtering of multiple DNaseI Hypersensitivity Uniform Peaks tracks in an attempt to summarize multiple individual experiments, which also can be viewed individually in the browser. Example of how the Transcription Factor ChIP Track is generated from multiple experiments [] This session helps illustrate the process behind creating the TFBS Clusters track. Uniform processing resulted in a comparable signal scores viewable in the wgEncodeAwgTfbsUniform track, that was then used to generate the clustered score in the wgEncodeRegTfbsClusteredV3 track, where a normalization factor was used to attempt to better distribute scores evenly. The session originated from a question regarding whether it was correct to interpret the darker score as increased biological evidence of binding for a transcription factor at a particular spot. The response shared how by filtering to display only the JUN, JUNB, JUND, and MYC factors you can see several individual "Uniform ...c-Myc" tracks displayed below the clusters track. Those are the separate wgEncodeAwgTfbsUniform tracks used to generate the processed clustered summary wgEncodeRegTfbsClusteredV3 track for this MYC cluster. Those individual uniform processed scores were used to create the cluster score given to the MYC cluster. Like the MYC factor, you can also click the JUN factors and you will see there is only one observed cell type where this data indicates this factor binds at this location. And similarly below, you will see the "Uniform... Jun" tracks that contributed to the clusters track. Mailing list support Example of displaying repeats in the hg19 Assembly [] A user was interested in an intron of the gene PHKB where a repeat element for AluY was given. The user could not find the Alu repeat, and the response was to aid the user in turning on the RepeatMasker track where information on repeats, such as the AluY element desired, is contained. Example of refGene exonFrames coming from mRNA [] A user had written some code to generate exon frame data from the refGene table, based on the belief that the exonFrames should be possible to derive entirely from the cds{Start,End} and exon{Starts,Ends} by taking differences and dividing by three for the size of codons. However, the user noticed an inconsistency with the their generated exonFrames output compared to the information in the refGene table. In the response the explanation was that the exon frames come from the mRNA, not the genome, and the example provided (NM_001282171/KIR3DS1) represented a transcript where there were deletions in respect to the reference at chr19_KI270922v1_alt:123,732-123,733. aacaga..agtgaacagc 000870 |||||| |||||||||| >>>>>> aacagaacagtgaacagc 123743 By clicking into the details of the RefSeq gene NM_001282171 transcript and clicking the first link in the section titled "mRNA/Genome Alignments" one can see the alignments of the mRNA to the genome and find the dots aga..agt indicating a deletion. More information about sessions What is a session? A session is a specific set of track combinations, that can include custom tracks allowing snapshots of browser activities. Multiple sessions may be saved for future reference, for comparing different data sets, or for sharing with your colleagues. Follow these links to learn how to create and share sessions. What is the session gallery? The Session Gallery is a collection of track views that help highlight viewing different topics in the browser. The sessions in the Session Gallery were created in the browser and then saved to a local file, which was then uploaded to an online location. This allows creating a single link, such as http://genome.ucsc.edu/cgi-bin/hgTracks?hgS_doLoadUrl=submit&hgS_loadUrlName=U, where U is the URL of the session file, e.g. http://www.mysite.edu/~me/mySession.txt, enabling users to maintain external control of the content file for easy update. Public Sessions - make a Session Gallery for your laboratory! After the creation of this Session Gallery page, UCSC created short links to sessions and released our Public Sessions page. Public Sessions allows users to publish their sessions and annotate them with a description. By using the search=searchTerm feature, it is now possible for groups to create their own laboratory-focused Session Gallery, by selecting sessions to add to the Public Sessions collection and describing their data with common unique terms. For example, a link to the Public Sessions page such as http://genome.ucsc.edu/cgi-bin/hgPublicSessions?search=protein will search all Public Sessions with "protein" mentioned in their description. If "protein" were replaced with "YourLabName" you could then have a link to display a gallery of sessions related to your laboratory data. Another example is http://genome.ucsc.edu/cgi-bin/hgPublicSessions?search=sessionView, which uses the unique term "sessionView" to collect related sessions. Read more in this blog post. Don't copy that link -- make a Session! Please see the How to share your UCSC screenthoughts and Sharing Data with Sessions and URLs blog posts for a discussion about sessions. Most new users make the mistake of sharing a URL of the current view to others for collaboration. This can lead to many problems as each URL is given a unique identifier called a "hgsid" that allows users to continue actively manipulating their data across different platforms (FireFox,Chrome,IE). The person one shares the link with will be interpreted as the same user and will have the power to continue manipulating the original shared content. When you want to share information in the browser in a more permanent fashion you should create a session and share the resulting URL. From the Session page, copy the link that says "Browser" with a right-click. The URL should look something like the following: http://genome.ucsc.edu/cgi-bin/hgTracks?hgS_doOtherUser=submit&hgS_otherUserName=sessionGallery&hgS_otherUserSessionName=hg19_watsonKriek In the above case a user name was created called sessionGallery and a session was created called hg19_watsonKriek. This more stable link is the best way to share information, with the only further improvement being to locally download your session file and then remotely share it from your own private collection in an internet accessible location using the hgS_loadUrlName= option. For example seen in this link: http://genome.ucsc.edu/cgi-bin/hgTracks?hgS_doLoadUrl=submit&hgS_loadUrlName=http://genome.ucsc.edu/goldenPath/help/examples/sessions/encodeDemonstration Custom Track caveats We have changed our policy about the persistence of session data including custom tracks in sessions. In the past saved sessions were only committed to persist for four months after the last access and custom tracks in sessions were subject to persist for at least 48 hours after the last time they were viewed. We have now moved to not remove session data, unless deleted, and to not remove custom tracks in sessions. However, please note that the UCSC Genome Browser is not a data storage service, so you will want to keep backups of all your custom tracks on a local machine, just in case there is a possible loss of data due to unforeseen circumstances. For mirrors, when loading sessions that contain custom data from custom tracks or hubs an issue can arise. If you create a session on our genome-euro or genome-asia machines with custom track data, and then try to load that session on our machine in the US, the custom track data will not populate through only the saved session file. The session only points to where the custom track data exists on a machine, and if loaded on a new machine that does not have the custom track in the same location, the custom track data will not be available to display. One fix for this issue is to load your custom tracks on both machines, then the sessions should find the data on each machine. A similar issue can occur with hubs, where different machines will have different prefixes for each hub, and so sessions on different machines will call the hubs by different unique identifiers. You can read some more information about these differences in sessions here. Lastly, please note that track data from BLAT results will not persist like session custom tracks. If you perform a BLAT search and create a session to keep those BLAT results, the BLAT display will not persist. However, if you wish to make BLAT data remain, you can set the BLAT results to "Output type: psl no header" and then paste the results into the custom track page, perhaps giving a top line such as track name="blat" description="My BLAT results for X,Y,Z" to describe it. 
/goldenPath/help/cloud.html:Cloud_Data_and_Software_Resources	Cloud Data and Software Resources The UCSC Genome Browser aims to support researchers operating in the cloud. Topics such as data access on the cloud, including details about the API and Amazon s3://genome-browser bucket, software installations for cloud computing, and references to helpful tools are discussed on this page. Contents Cloud Data Amazon S3 - What is the Amazon S3 genome-browser bucket? - What specific files are in the s3://genome-browser bucket? - How can one get data from the s3://genome-browser bucket? REST API - What is the REST API? - What kind of data can you get from the REST API? Download Server - What is the download server and how does one use it? MySQL Server - What is the MySQL server and how does one use it? Cloud Software GBiB/GBiC - What are GBiB and GBiC? (Genome Browser in a Box/in the Cloud) Docker - Do you support Docker? Helpful Tools Browser Utilities - How do I extract data from the bigBed/2bit data formats? Amazon Ecosystem - Where can I learn more about Amazon Tools? What is the Amazon s3://genome-browser bucket? S3 stands for Simple Storage Service, and it is the name for cloud storage in Amazon Web Services (AWS). The data available through S3 is essentially stored in a folder called a bucket, and files are called objects. The s3://genome-browser bucket is a copy of the main data available on our UCSC Genome Browser Download website: https://hgdownload.soe.ucsc.edu/downloads.html By placing our Download server files in an S3 bucket, developers working in the cloud can more easily integrate with UCSC data. You can learn more about how S3-object-based storage works, and its advantages of being accessible anywhere across the world with low latency and high durability by reviewing Amazon's S3 documentation. What specific files are in the s3://genome-browser bucket? The data mirrors our UCSC Genome Browser Download website's main rsync directories: UCSC Human Golden Path Downloads s3://genome-browser/goldenPath UCSC Human Genome Browser Gbdb Data Files s3://genome-browser/gbdb UCSC Human Genome Raw Mysql Tables s3://genome-browser/mysql UCSC Human Genome Web Site CGI Binaries s3://genome-browser/cgi-bin UCSC Human Genome Web Site Htdocs s3://genome-browser/htdocs - The goldenPath directory is organized by assembly name, and represents the file structure on our Download server, which includes README.txt files. For instance, the sequence data for the human hg38 assembly would be found in this location with an instructive README.txt: goldenPath/hg38/bigZips/README.txt. The README.txt, also available on the Download website, informs that the most recent patch-inclusive sequence is found in goldenPath/hg38/bigZips/latest/. - The gbdb directory, also organized by assembly name, provides access to genome browser database files in binary format used by the browser software. For instance, the underlying binary indexed sequence data for the hg38 databases used in the display in the UCSC Genome Browser would be located in the following location, gbdb/hg38/hg38.2bit, matching the file in the goldenPath/hg38/bigZips/latest/ directory, reflecting how these files are operated on by the UCSC Genome Browser software in order to display assembly sequence when browsing. - The mysql directory, also organized by assembly name, provides access to MySQL database tableName.MYD files, and their related tableName.MYI index and tableName.frm format files, providing a copy of the tables used by the main Browser site. - The cgi-bin directory is a copy of the software run on the main browser site. - The htdocs directory is a copy of the html pages used on the main browser site, such as htdocs/goldenPath/pubs.html which lists our publications. How can one get data from the s3://genome-browser bucket? Amazon provides an AWS Command Line Interface (AWS CLI) which includes options such as sync. Here is an example to download an AWS bucket with CLI: aws s3 sync s3://bucket-name . The data is also available via http at genome-browser.s3-website-us-east-1.amazonaws.com where files can be accessed. Examples - For instance, here is an example of accessing a README file in the goldenPath/Downloads directory: http://genome-browser.s3-website-us-east-1.amazonaws.com/goldenPath/hg38/bigZips/README.txt - And here is an example link that would access the gbdb/ binary data directory for the human hg38 assembly 2bit file: http://genome-browser.s3-website-us-east-1.amazonaws.com/gbdb/hg38/hg38.2bit - And here is an example link that would access our publications html page from the bucket's htdocs/ hypertext document directory: http://genome-browser.s3-website-us-east-1.amazonaws.com/htdocs/goldenPath/pubs.html What is the REST API? The UCSC Genome Browser has a REST API for the programmatic extraction of data. REST is an acronym for REpresentational State Transfer and API stands for Application Programming Interface, read more on the help page: http://genome.ucsc.edu/goldenPath/help/api.html The REST API returns data in JavaScript Object Notation (JSON) format, which can easily be sent between computers, and used by many different programming languages. Data can be accessed with this URL: https://api.genome.ucsc.edu/ By adding different endpoint functions such as /list/ or /getData/ specific results can be obtained. Examples wget -O- 'https://api.genome.ucsc.edu/list/publicHubs' wget -O- 'https://api.genome.ucsc.edu/getData/sequence?genome=hg38;chrom=chrM;start=4321;end=5678' What kind of data can you get from the REST API? With different endpoint functions such as /list/ or /getData/ URLs can be constructed to pull specific results. Endpoint function Required Optional ------------------- ------------------------------------------- ---------------------------------------------------------- /list/publicHubs (none) (none) /list/ucscGenomes (none) (none) /list/hubGenomes hubUrl (none) /list/tracks genome or (hubUrl and genome) trackLeavesOnly=1 /list/chromosomes genome or (hubUrl and genome) track /list/schema (genome or (hubUrl and genome)) and track (none) /getData/sequence (genome or (hubUrl and genome)) and chrom start and end /getData/track (genome or (hubUrl and genome)) and track chrom, (start and end), maxItemsOutput, jsonOutputArrays By reviewing example data access URLs demonstrating of list and getData functions and further practical examples URLs of extracting specific track data items you can learn more about the ways of using the API to extract data. What is the Download server and how does one use it? The UCSC Genome Browser Download website, hgdownload.soe.ucsc.edu, is the source of the data hosted in the Amazon s3://genome-browser bucket. It can be viewed in a web browser to access specific download files, or the data can be copied with rysnc commands. Examples For instance, the following rsync command will show you the various rysnc directories available on our Download server: $ rsync -a -P rsync://hgdownload.soe.ucsc.edu/ genome UCSC Human Genome Downloads sars UCSC Human Genome SARS Downloads htdocs UCSC Human Genome Web Site Htdocs goldenPath UCSC Human Golden Path Downloads cgi-bin UCSC Human Genome Web Site CGI Binaries x86_64 cgi-bin-i386 UCSC Human Genome Web Site CGI Binaries i386 gbdb UCSC Human Genome Browser Gbdb Config Files archives UCSC Human Genome Browser Archived Config Files mysql UCSC Human Genome Raw Mysql Tables gbib UCSC Genome Browser in a Box hubs UCSC Genome Browser Public Hubs - For instance, here is an example of accessing a README file in the goldenPath/Downloads directory: rsync -a -P rsync://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/README.txt ./ - And here is an example link that would access the gbdb/ binary data directory for the human hg38 assembly 2bit file: rsync -a -P rsync://hgdownload.soe.ucsc.edu/gbdb/hg38/hg38.2bit ./ - And here is an example link that would access our publications html page from the bucket's htdocs/ hypertext document directory: rsync -a -P rsync://hgdownload.soe.ucsc.edu/htdocs/goldenPath/pubs.html ./ Many of these rsync directories exist to support the Genome Browser in a Cloud (GBiC) and the Genome Browser in a Box (GBiB) software products discussed below. Also note that there is a mirror of the download server available in Europe so the above rysnc commands can also be pointed to the hgdownload-euro locations. - For instance here is a command to access data from the Europe location: rsync -a -P rsync://hgdownload-euro.soe.ucsc.edu/gbdb/hg38/hg38.2bit ./ What is the MySQL server and how does one use it? The UCSC Genome Browser uses MariaDB (fork of MySQL) as the backend database server and maintains a public server at genome-mysql.soe.ucsc.edu to allow direct queries. Examples - For instance, here is an example of accessing the hg38 human assembly database and selecting from the table trackDb all the entries in the group (grp) "genes" and ordering those entries by tableName: mysql -h genome-mysql.soe.ucsc.edu -u genome -NBe 'select tableName from trackDb where grp = "genes" order by tableName' hg38 - And here is an example of accessing a specific Transcription Factor Binding Site (TFBS) table wgEncodeRegTfbsClusteredV3 on the human hg19 assembly and selecting entries from a 500 base pair region on chr1: mysql --user=genome --host=genome-mysql.soe.ucsc.edu -A -Ne 'select chrom,chromStart,chromEnd,name,score from wgEncodeRegTfbsClusteredV3 where chrom = "chr1" and chromStart > 10000 and chromEnd < 10500;' hg19 - And here is an example query that will pull all the long non-coding entries (lncRNA) from the wgEncodeGencodeBasicV39 table on the hg38 genome: mysql -u genome -h genome-mysql.soe.ucsc.edu hg38 -e 'select g.name,a.transcriptType from wgEncodeGencodeBasicV39 g, wgEncodeGencodeAttrsV39 a where (g.name = a.transcriptId) and (a.transcriptType = "lncRNA");' See the Downloading Data using MariaDB (MySQL) for more information. Also, there is a mirror of the MariaDb server available in Europe so commands can also be pointed to the genome-euro-mysql location. - For instance here is a command to access hg38 data from the Europe location: mysql -h genome-mysql-euro.soe.ucsc.edu -u genome -NBe 'show tables' hg38 What are GBiB and GBIC? (Genome Browser in a Box/in the Cloud) To replicate, or mirror, the software of the UCSC Genome Browser in another location we offer the Genome Browser in a Cloud (GBiC) and the Genome Browser in a Box (GBiB) software products. The GBiC is an installation script that automates the setup of a UCSC Genome Browser mirror including setting up MariaDB and Apache servers. The program downloads and configures MySQL and Apache, and then downloads the UCSC Genome Browser software to /usr/local/apache to make a local instance of the Browser. The GBiB is a small virtual machine version of the UCSC Genome Browser that can be run on a laptop or desktop computer. It requires an installation of a compatible version of the VirtualBox Software, and will then access annotation data on demand through the Internet from UCSC as used, or selective data can be downloaded for faster access. The GBiB and GBiC software tools resource the Download server to rsync data, as well as in certain circumstances the MySQL server to extract coordinate-specific table data. See the individual support pages for the GBiC and the GBiB for detailed information about how to install and operate both. You can get either the GBiC or the GBiB from the UCSC Genome Browser store free for non-commercial use. Do you support Docker? We do support a Dockerfile, that in essence points to the GBiC installation script. While we recommend our GBiC script, we understand many people are more familiar with working through Docker and provide Docker installation instructions. Please note, similar to how our GBiB and GBiC are available in the UCSC Genome Browser store, where usage of our mirror software is free for non-commercial use. Any commercial usage, including through the Docker image, involves a license. How do I extract data from the bigBed/2bit data formats? A lot of our data is stored in a binary indexed version called bigBed. This format saves space and also allows the extraction of information based on the first three fields (chrom, chromStart, chromEnd), which define annotation coordinate location. To pull information out of bigBed files there is a tool called bigBedToBed. By running the command by itself you can see the command options. bigBedToBed v1 - Convert from bigBed to ascii bed format. usage: bigBedToBed input.bb output.bed options: -chrom=chr1 - if set restrict output to given chromosome -start=N - if set, restrict output to only that over start -end=N - if set, restrict output to only that under end -bed=in.bed - restrict output to all regions in a BED file -udcDir=/dir/to/cache - place to put cache for remote bigBed/bigWigs -header - output a autoSql-style header (starts with '#'). Another similar tool is available to extract data from the binary indexed 2bit sequence storage format. The tool twoBitToFa can be given coordinate ranges and the DNA can be extracted from the file. Examples - For instance, here is an example of accessing the hg38 2bit human assembly sequence file hosted at the s3 Amazon bucket and extracting a small coordinate range: twoBitToFa -seq=chr1 -start=1234500 -end=1234600 http://genome-browser.s3-website-us-east-1.amazonaws.com/gbdb/hg38/hg38.2bit stdout >chr1:1234500-1234600 GCGTCCCTAGGTCAGGCCGTTGAGTTCGAGCTCCGATGGGCCACCTTGAA TCCAGGACTGACCGCCCGTGTGTGCACAGTTTGTTCTTGGACGAGGACTC - And here is an example of accessing the ENCODE Candidate Cis-Regulatory Elements (cCREs) bigBed file hosted on the Amazon s3 bucket and extracting enhancers in a defined region. bigBedToBed -chrom=chr1 -start=190000 -end=200000 http://genome-browser.s3-website-us-east-1.amazonaws.com/gbdb/hg38/encode3/ccre/encodeCcreCombined.bb stdout | head chr1 190865 191071 EH38E1310154 179 . 190865 191071 255,205,0 dELS,CTCF-boundd ELS 1.79282201562 enhDE1310154 EH38E1310154 distal enhancer-like signature Where can I learn more about Amazon Tools? The Amazon Ecosystem comes integrated with a collection of systems such as CloudFront, CloudWatch, Relational Database Service (RDS), Elastic Block Store (EBS), Lambda, and Aurora. Amazon Aurora is a MySQL and PostgreSQL-compatible relational database built for the cloud. The UCSC Genome Browser's tableName.MYD and tableName.MYI files can be used with Aurora, instead of installing MariaDb, however, there may be some services costs in Amazon for using Aurora. 
/goldenPath/help/hubQuickStart.html:Track_Hub_Quick_Start	Basic Track Hub Quick Start Guide Hubs are a method of displaying remote custom tracks quickly (binary indexed bigBed, bigWig, BAM or VCF formats), while providing more persistence and flexibility than normal custom tracks for any UCSC assembly (or remotely-hosted assembly in twoBit format). STEP 1: In a publicly-accessible directory, copy the hub.txt, genomes.txt, trackDb.txt files using the following command: wget -r --no-parent --reject "index.html*" -nH --cut-dirs=3 http://genome.ucsc.edu/goldenPath/help/examples/hubDirectory/ Alternatively, if you do not have wget installed, use curl: curl -O http://genome.ucsc.edu/goldenPath/help/examples/hubDirectory/hub.txt curl -O http://genome.ucsc.edu/goldenPath/help/examples/hubDirectory/genomes.txt mkdir hg19 cd hg19 curl -O http://genome.ucsc.edu/goldenPath/help/examples/hubDirectory/hg19/trackDb.txt If you do not have curl, you can alternatively use a text editor and directly recreate the above three files. Note: there is now a useOneFile on hub setting that allows the hub properties to be specified in a single file. More information about this setting can be found on the Genome Browser User Guide. STEP 2: Paste your hub.txt link (http://yourURL/hub.txt) into the Connected Hubs tab of the Track Data Hubs page, then click the "Use Selected Hubs" button. It should work the same as pasting the original hub.txt file: http://genome.ucsc.edu/goldenPath/help/examples/hubDirectory/hub.txt STEP 3: Congratulations! Your hub should display! If you are not already browsing the hg19 assembly on chr21, change assemblies and navigate to the gene SOD1 to see data displayed for each of the hub's BAM, bigWig, bigBed, and VCF tracks. If you are having problems, be sure all your files and the hg19 directory are publicly-accessible. For hubs to work, your server must also accept byte-ranges. You can check using the following command to verify "Accept-Ranges: bytes" displays: curl -I http://yourURL/hub.txt The three parts of a hub Now that you have a working hub copied from above, you can edit the three main text documents to get an idea of how they work, and also point to your local files. - Understanding hub.txt - Understanding genomes.txt - Understanding trackDb.txt [Example hub directory] Example Hub Directory Hubs begin with the short hub.txt, which describes your hub and informs the UCSC Browser where to find the underlying assemblies by defining the location of genomes.txt. The genomes.txt file outlines the hub's assemblies and defines the location of each assembly's trackDb.txt, which is basically the heart of your hub, defining the location of all the binary indexed track data for each assembly. The hub.txt file can be as short as four lines, and the genomes.txt as short as two. The trackDb.txt file is typically much larger. At its most basic, it defines the tracks in the hub using stanzas about the type, location, and name of the binary files to display. However, it provides real power to tailor your presentation using additional trackDb settings. Note that the Browser waits 5 minutes before checking for any changes to these files. When editing hub.txt, genomes.txt,and trackDb.txt, you can shorten this delay by adding udcTimeout=1 to your URL. For more information, see the Debugging and Updating Track Hubs section of the Track Hub User Guide. For more detailed instructions on setting up a hub, refer to the Setting Up Your Own Track Hub section of the Track Hub User Guide. Resources - Track Hub User Guide - Track Database (trackDb) Definition Document - Assembly Hubs Wiki - Public Hub Guidelines - Quick Start Guide to Organizing Track Hubs into Groupings - Quick Start Guide to Assembly Track Hubs - Quick Start Guide to Searchable Track Hubs Understanding hub.txt hub MyHubsNameWithoutSpaces shortLabel My Hub's Name longLabel Name up to 80 characters versus shortLabel limited to 17 characters genomesFile genomes.txt email myEmail@address descriptionUrl aboutMyHub.html A hub starts with a few short lines in hubs.txt. The hub.txt file informs the UCSC Browser where to find the underlying assemblies via the genomesFilegenomes.txt parameter, which in turn will direct the Browser to each assembly's related binary indexed track data outlined in the trackDb.txt. The optional descriptionUrl field allows you to add an HTML page describing your hub. See an example of a basic hub.txt file here. Understanding genomes.txt genome assembly_database_1 trackDb assembly_1_path/trackDb.txt genome assembly_database_2 trackDb assembly_2_path/trackDb.txt The genomes.txt text file can be as short as a two-line stanza when using only one UCSC assembly (e.g., genome panTro4 and trackDb panTro4/trackDb.txt). The genome line directs the Browser to use the panTro4 chimp genome, while the trackDb line points the Browser to the associated trackDb.txt, which will outline all of the assembly's tracks. View an example of a basic genomes.txt file here. Understanding trackDb.txt track uniqueNameNoSpacesOrDots type track_type bigDataUrl track_data_url shortLabel label 17 chars longLabel long label up to 80 chars The trackDb.txt file uses stanzas for each track to inform the Browser of the name, type, location, and description of each binary file to display. The trackDb settings allow further display control such as by adding the line color255,0,0 to define a track's color. Each track type (bigBed, bigWig, bam,and vcfTabix) has further customizable trackDb settings. The bigDataUrl can be a relative path to a local file or a publicly-accessibly URL that accepts byte-ranges. The trackDb.txt also allows for advanced track grouping known as composites, superTracks and multiWigs. See a basic example trackDb.txt. When type is set to bigBed, the track hub assumes that the bigBed track is BED3 by default. To allow track hubs to use all fields in the bigBed file, one must define how many columns to expect. For example, if a bigBed file has nine columns, which would include an itemRgb field to display a R,G,B color value (e.g. 255,0,0), specify the type as type bigBed 9. If the file contains the first 9 standard BED columns, one could use type bigBed 9 + to denote the additional non-standard columns as defined in AutoSql format(.as) file, -as=bedExample1.as. 
/goldenPath/help/hgCodonColoring.html:Genome_Browser_Codon_Coloring	Coloring Gene Predictions and Annotations by Codon The Genome Browser's codon coloring feature allows users to quickly validate and compare gene predictions and annotations. To turn on codon coloring, select the genomic codons option from the Color track by codons pull-down menu. Color Legend - bright green codons/lines: methionine, including start codons - cyan text: partial codon, either spliced or truncated - red "*": stop codons - black "X": error (e.g. truncated last codon) All other codons are shown in the original track color, with codon boundaries indicated by alternating lighter and darker shades. Details Each codon is colored and labeled in the direction of transcription. This coloring applies only to gene prediction and annotation tracks; thus the codon sequence is colored in reference to the genome and not the associated mRNA. A gene prediction without a CDS will not be colored; these can be non-protein-coding RNAs or predictions where a CDS annotation was not provided in the original data. To view codon coloring, the option must be turned on, the track must be in squish, pack, or full display mode, and the browser must be zoomed-in to a level at which there is at least one pixel per codon. When zoomed out to large regions of the genome, the browser will not display codon coloring. To view labeling, the track must be zoomed to within 3 times the base level. 
/goldenPath/help/twoBit.html:Genome_Browser_TwoBit_Sequence	TwoBit Sequence Archives A twoBit file is a highly efficient way to store genomic sequence. The format is defined here. Note that lower-case nucleotides are considered masked in twoBit, which can cause such sequence to be ignored when using the -mask option with gfServer; therefore, you may wish to convert lower-case sequence to upper-case when preparing the FASTA format. To complete the steps below you must first download the faToTwoBit, twoBitInfo, and twoBitToFa utilities. For more information on downloading our command-line utilities, see these instructions. To create a twoBit file, follow these steps: 1. Prepare the sequence for your twoBit file in a FASTA-formatted file (i.e. genome.fa). 2. Run the faToTwoBit program on your FASTA file: faToTwoBit genome.fa genome.2bit 3. Use twoBitInfo to verify the sequences in this assembly and create a chrom.sizes file, which is useful to construct the big* files in later processing steps: twoBitInfo genome.2bit stdout | sort -k2rn > genome.chrom.sizes The twoBit commands can function with the .2bit file as a URL: twoBitInfo -udcDir=. http://your-website.edu/~user/genome.2bit | sort -k2nr > genome.chrom.sizes Sequence can be extracted from the .2bit file with the twoBitToFa command, for example: twoBitToFa -seq=chr1 -udcDir=. http://your-website.edu/~user/genome.2bit stdout > genome.chr1.fa Examples of extracting sequences See these series of blog posts about Accessing the Genome Browser Programmatically to see examples of extracting sequences remotely, such as the following: $ twoBitToFa http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.2bit:chr1:100100-100200 stdout >chr1:100100-100200 gcctagtacagactctccctgcagatgaaattatatgggatgctaaatta taatgagaacaatgtttggtgagccaaaactacaacaagggaagctaatt Also, see the API getData functions to see examples of using the URL, such as the following: https://api.genome.ucsc.edu/getData/sequence?genome=hg38;chrom=chr1;start=100100;end=100200 downloadTime: "2022:05:19T18:45:56Z" downloadTimeStamp: 1652985956 genome: "hg38" chrom: "chr1" start: 100100 end: 100200 dna: "gcctagtacagactctccctgcagatgaaattatatgggatgctaaattataatgagaacaatgtttggtgagccaaaactacaacaagggaagctaatt" 
/goldenPath/help/bigPsl.html:Genome_Browser_bigPsl_Track_Format	bigPsl Track Format The bigPsl format stores alignments between two sequences just as PSL files do; however, bigPsl files are compressed and indexed as bigBeds. PSL files are converted to bigPsl files using the program bedToBigBed, run with the -as option to pull in a special autoSql (.as) file that defines the fields of the bigPsl. The bigPsl files are in an indexed binary format. The main advantage of this format is that only those portions of the file needed to display a particular region are transferred to the Genome Browser server. Because of this, bigPsl files have considerably faster display performance than regular PSL files when working with large data sets. The bigPsl file remains on your local web-accessible server (http, https or ftp), not on the UCSC server, and only the portion needed for the currently displayed chromosomal position is locally cached as a "sparse file". If you do not have access to a web-accessible server and need hosting space for your bigPsl files, please see the Hosting section of the Track Hub Help documentation. bigPSL format definition The following autoSql definition is used to specify bigPsl alignment files. This definition, contained in the file bigPsl.as, is pulled in when the bedToBigBed utility is run with the -as=bigPsl.as option. table bigPsl "bigPsl pairwise alignment" ( string chrom; "Reference sequence chromosome or scaffold" uint chromStart; "Start position in chromosome" uint chromEnd; "End position in chromosome" string name; "Name or ID of item, ideally both human readable and unique" uint score; "Score (0-1000)" char[1] strand; "+ or - indicates whether the query aligns to the + or - strand on the reference" uint thickStart; "Start of where display should be thick (start codon)" uint thickEnd; "End of where display should be thick (stop codon)" uint reserved; "RGB value (use R,G,B string in input file)" int blockCount; "Number of blocks" int[blockCount] blockSizes; "Comma separated list of block sizes" int[blockCount] chromStarts;"Start positions relative to chromStart" uint oChromStart; "Start position in other chromosome" uint oChromEnd; "End position in other chromosome" char[1] oStrand; "+ or -, - means that psl was reversed into BED-compatible coordinates" uint oChromSize; "Size of other chromosome." int[blockCount] oChromStarts;"Start positions relative to oChromStart or from oChromStart+oChromSize depending on strand" lstring oSequence; "Sequence on other chrom (or empty)" string oCDS; "CDS in NCBI format" uint chromSize; "Size of target chromosome" uint match; "Number of bases matched." uint misMatch; "Number of bases that don't match " uint repMatch; "Number of bases that match but are part of repeats " uint nCount; "Number of 'N' bases " uint seqType; "0=empty, 1=nucleotide, 2=amino_acid" ) The value of the oStrand field indicates whether or not the stored psl data should be reverse-complemented before it is outputted or displayed. This is necessary because the bigPsl file stores reference coordinates on the positive strand, as required by the BED format. The strand field indicates whether the positions in oChromStarts are listed from the chromosome beginning (+) or end (-). Additional fields: Since a bigPsl file is a bigBed file, additional fields can be added as bigBed fields. The additional bigBed fields are defined after the seqType field of the bigPsl.as file. See Example 3 of the bigBed Track Format page for an example on how to create a bigBed file with extra (custom) fields. The additional fields can be used for custom mouseOvers, feature filters, and coloring options. Contact us at genome-www@soe.ucsc.edu if you run into difficulty using them. The bedToBigBed utility uses a substantial amount of memory: approximately 25% more RAM than the uncompressed BED input file. The -maxAllow option to bedToBigBed can be used to allow the tool to use more than 16GB of RAM. Creating a bigPsl track To create a bigPsl track, follow these steps: Step 1. If you already have a PSL file created using BLAT or another tool, skip to Step 2. Otherwise, download the example PSL file bigPsl.psl for the human GRCh38/hg38 assembly. You will also want to download the files bigPsl.fa and bigPsl.cds if you would like to use the alternate options described in Step 4, below. Step 2. Download the bedToBigBed and pslToBigPsl programs from the binary utilities directory. Step 3. Use the fetchChromSizes script from the same directory to create a chrom.sizes file for the UCSC database with which you are working (e.g., hg38). Alternatively, you can download the chrom.sizes file for any assembly hosted at UCSC from our downloads page (click on "Full data set" for any assembly). For example, the hg38.chrom.sizes file for the hg38 database is located at http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.chrom.sizes. Step 4. Use the pslToBigPsl utility to create a bigPsl file in bed12+13 format that contains the 25 fields described in the bigPsl format definition above. The file must include the 13 extra fields: oChromStart, oChromEnd, oStrand, oChromSize, oChromStarts, oSequence, oCDS, chromSize, match, misMatch, repMatch, nCount, and seqType. pslToBigPsl bigPsl.psl stdout | sort -k1,1 -k2,2n > bigPsl.txt If you are using your own PSL file, you may have corresponding FASTA and CDS files that accompany it. You can provide these files as input to pslToBigPsl to generate a more informative bigPsl file: pslToBigPsl bigPsl.psl -cds=bigPsl.cds -fa=bigPsl.fa stdout | sort -k1,1 -k2,2n > bigPsl.txt Step 5. Create the binary indexed bigPsl file from your sorted bigPsl input file using the bedToBigBed utility: bedToBigBed -as=bigPsl.as -type=bed12+13 -tab bigPsl.txt chrom.sizes bigPsl.bb Step 6. Move the newly created bigPsl file (bigPsl.bb) to a web-accessible http, https, or ftp location. Step 7. Construct a custom track using a single track line. Note that any of the track attributes listed here are applicable to tracks of type bigBed. The most basic version of the track line will look something like this: track type=bigPsl name="My Big Psl" description="Some mRNAs Discovered from Data from My Lab" bigDataUrl=http://myorg.edu/mylab/myBigPsl.bb Step 8. Paste the custom track line into the text box on the custom track management page. The bedToBigBed program can be run with several additional options. For a full list of the available options, type bedToBigBed (with no arguments) on the command line to display the usage message. Examples Example #1 In this example, you will create a bigPsl custom track using an existing bigPsl file, bigPsl.bb, located on the UCSC Genome Browser http server. This file contains data for the hg38 assembly. To create a custom track using this bigPsl file: 1. Construct a track line that references the file: track type=bigPsl name="bigPsl Example One" description="A bigPsl file" bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigPsl.bb 2. Paste the track line into the custom track management page for the human assembly hg38. 3. Click the "submit" button. Custom tracks can also be loaded via one URL line. The link below loads the same bigPsl track, and sets additional display parameters in the URL: http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&position=chr1:10000-200000&hgct_customText=track%20type=bigPsl%20name=Example%20bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigPsl.bb%20visibility=pack After this example bigPsl is loaded in the Genome Browser, click on a track item in the Genome Browser's track display. Note that the details page displays information about the alignment, similar that which is available for PSL tracks, as well as links that display the browser position of the alignment and other detailed information about the alignment. Example #2 In this example, you will create your own bigPsl file from an existing bigPsl input file. 1. Save the example bed12+13 file bigPsl.txt to your computer (Step 4 in Creating a bigPsl track, above). 2. Download the bedToBigBed utility (Step 2, above). 3. Save the hg38.chrom.sizes text file to your computer. This file contains the chrom.sizes for the human (hg38) assembly (Step 3, above). 4. Save the autoSql file bigPsl.as to your computer. 5. Run the bedToBigBed utility to create the bigPsl output file (Step 5, above): bedToBigBed -as=bigPsl.as -type=bed12+13 -tab bigPsl.txt hg38.chrom.sizes bigPsl.bb 6. Place the newly created bigPsl file (bigPsl.bb) on a web-accessible server (Step 6, above). 7. Construct a track line that points to the bigPsl file (Step 7, above). 8. Create the custom track on the human assembly hg38 (Dec. 2013), and view it in the Genome Browser (Step 8, above). Sharing your data with others If you would like to share your bigPsl data track with a colleague, learn how to create a URL by looking at Example #6 on this page. Extracting data from the bigPsl format Because bigPsl files are an extension of bigBed files, which are indexed binary files, it can be difficult to extract data from them. UCSC has developed the following programs to assist in working with bigBed formats, available from the binary utilities directory. - bigBedToBed — converts a bigBed file to ASCII BED format. - bigBedSummary — extracts summary information from a bigBed file. - bigBedInfo — prints out information about a bigBed file. As with all UCSC Genome Browser programs, simply type the program name (with no parameters) at the command line to view the usage statement. Troubleshooting If you encounter an error when you run the bedToBigBed program, check your input file for data coordinates that extend past the the end of the chromosome. If these are present, run the bedClip program (available here) to remove the problematic row(s) in your input file before running the bedToBigBed program. 
/goldenPath/help/hgTracksHelp.html:Genome_Browser_User's_Guide	Genome Browser User Guide Contents What does the Genome Browser do? Configuring the Genome Browser display Annotation track descriptions Using BLAT alignments Getting Started with: - Genome Browser Gateway - Table Browser - Using Sessions - Gene and Pathway Viewer - Genome Graphs DNA text formatting Converting data between assemblies Downloading genome data Creating and managing custom annotation tracks Getting started on Track Hubs Hub track database definitions Track Hubs for GenArk assembly hub Track Hubs in a single file Using the VisiGene Image Browser Search the Genome Browser help pages:   Search the entire Genome Browser website:   Browse the Genome Browser mailing list. Questions and feedback are welcome. What does the Genome Browser do? As vertebrate genome sequences near completion and research re-focuses on their analysis, the issue of effective sequence display becomes critical: it is not helpful to have 3 billion letters of genomic DNA shown as plain text! As an alternative, the UCSC Genome Browser provides a rapid and reliable display of any requested portion of genomes at any scale, together with dozens of aligned annotation tracks (known genes, predicted genes, ESTs, mRNAs, CpG islands, assembly gaps and coverage, chromosomal bands, mouse homologies, and more). Half of the annotation tracks are computed at UCSC from publicly available sequence data. The remaining tracks are provided by collaborators worldwide. Users can also add their own custom tracks to the browser for educational or research purposes. The Genome Browser stacks annotation tracks beneath genome coordinate positions, allowing rapid visual correlation of different types of information. The user can look at a whole chromosome to get a feel for gene density, open a specific cytogenetic band to see a positionally mapped disease gene candidate, or zoom in to a particular gene to view its spliced ESTs and possible alternative splicing. The Genome Browser itself does not draw conclusions; rather, it collates all relevant information in one location, leaving the exploration and interpretation to the user. The Genome Browser supports text and sequence based searches that provide quick, precise access to any region of specific interest. Secondary links from individual entries within annotation tracks lead to sequence details and supplementary off-site databases. To control information overload, tracks need not be displayed in full. Tracks can be hidden, collapsed into a condensed or single-line display, or filtered according to the user's criteria. Zooming and scrolling controls help to narrow or broaden the displayed chromosomal range to focus on the exact region of interest. Clicking on an individual item within a track opens a details page containing a summary of properties and links to off-site repositories such as PubMed, GenBank, Entrez, and OMIM. The page provides item-specific information on position, cytoband, strand, data source, and encoded protein, mRNA, genomic sequence and alignment, as appropriate to the nature of the track. A blue navigation bar at the top of the browser provides links to several other tools and data sources. For instance, under the "View" menu, the "DNA" link enables the user to view the raw genomic DNA sequence for the coordinate range displayed in the browser window. This DNA can encode track features via elaborate text formatting options. Other links tie the Genome Browser to the BLAT alignment tool, provide access to the underlying relational database via the Table Browser, convert coordinates across different assembly dates, and open the window at the complementary Ensembl or NCBI Genome Data Viewer annotation. The browser data represents an immense collaborative effort involving thousands of people from the international biomedical research community. The UCSC Bioinformatics Group itself does no sequencing. Although it creates the majority of the annotation tracks in-house, the annotations are based on publicly available data contributed by many labs and research groups throughout the world. Several of the Genome Browser annotations are generated in collaboration with outside individuals or are contributed wholly by external research groups. UCSC's other major roles include building genome assemblies, creating the Genome Browser work environment, and serving it online. The majority of the sequence data, annotation tracks, and even software are in the public domain and are available for anyone to download. In addition to the Genome Browser, the UCSC Genome Bioinformatics group provides several other tools for viewing and interpreting genome data: - BLAT - a fast sequence-alignment tool similar to BLAST. Read more. - Table Browser - convenient text-based access to the database underlying the Genome Browser. Read more. - Genome Graphs - a tool that allows you to upload and display genome-wide data sets such as the results of genome-wide SNP association studies, linkage studies and homozygosity mapping. Read more. - Gene Sorter - expression, homology, and other information on groups of genes that can be related in many ways. Read more. Getting started: Genome Browser gateway The UCSC Genome Bioinformatics home page provides access to Genome Browsers on several different genome assemblies. To get started, click the Browser link on the blue sidebar. This will take you to a Gateway page where you can select which genome to display. Note that there are also official mirror sites in Europe and Asia for users who are geographically closer to those continents than to the western United States. Opening the Genome Browser at a specific position To get oriented in using the Genome Browser, try viewing a gene or region of the genome with which you are already familiar, or use the default position. To open the Genome Browser window: 1. Select the clade, genome and assembly that you wish to display from the corresponding pull-down menus. Assemblies are typically named by the first three characters of an organism's genus and species names. For older assemblies that are no longer available from the menu, the data may still be available on our Downloads page. 2. Specify the genome location you'd like the Genome Browser to open to. To select a location, enter a valid position query in the search term text box at the top of the Gateway page or accept the default position already displayed. The search supports several different types of queries: gene symbols, mRNA or EST accession numbers, chromosome bands, descriptive terms likely to occur in GenBank text, or specific chromosomal ranges. 3. Click the submit button to open up the Genome Browser window to the requested location. In cases where a specific term (accession, gene name, etc.) was queried, the item will be highlighted in the display. Occasionally the Gateway page returns a list of several matches in response to a search, rather than immediately displaying the Genome Browser window. When this occurs, click on the item in which you're interested and the Genome Browser will open to that location. The search mechanism is not a site-wide search engine. Instead, it primarily searches GenBank mRNA records whose text annotations can include gene names, gene symbols, journal title words, author names, and RefSeq mRNAs. Searches on other selected identifiers, such as NP and NM accession numbers, OMIM identifiers, and Entrez Gene IDs are supported. However, some types of queries will return an error, e.g. post-assembly GenBank entries, withdrawn gene names, and abandoned synonyms. If your initial query is unsuccessful, try entering a different related term that may produce the same location. For example, if a query on a gene symbol produces no results, try entering an mRNA accession, gene ID number, or descriptive words associated with the gene. []       Visit our Video Page. Visit our YouTube channel for more videos. Finding a genome location using BLAT If you have genomic, mRNA, or protein sequence, but don't know the name or the location to which it maps in the genome, the BLAT tool will rapidly locate the position by homology alignment, provided that the region has been sequenced. This search will find close members of the gene family, as well as assembly duplication artifacts. An entire set of query sequences can be looked up simultaneously when provided in fasta format. A successful BLAT search returns a list of one or more genome locations that match the input sequence. To view one of the alignments in the Genome Browser, click the browser link for the match. The details link can be used to preview the alignment to determine if it is of sufficient match quality to merit viewing in the Genome Browser. If too many BLAT hits occur, try narrowing the search by filtering the sequence in slow mode with RepeatMasker, then rerunning the BLAT search. For more information on conducting and fine-tuning BLAT searches, refer to the BLAT section of this document. Opening the Genome Browser with a custom annotation track You can open the Genome Browser window with a custom annotation track displayed by using the Add Custom Tracks feature available from the gateway and annotation tracks pages. For more information on creating and using custom annotation tracks, refer to the Creating custom annotation tracks section. Annotation track data can be entered in one of three ways: - Enter the file name for an annotation track source file in the Annotation File text box. - Type or paste the annotation track data into the large text box. - If the annotation data are accessible through a URL, enter the URL name in the large text box. Once you've entered the annotation information, click the submit button at the top of the Gateway page to open up the Genome Browser with the annotation track displayed. The Genome Browser also provides a collection of custom annotation tracks contributed by the UCSC Genome Bioinformatics group and the research community. NOTE: If an annotation track does not display correctly when you attempt to upload it, you may need to reset the Genome Browser to its default settings, then reload the track. For information on troubleshooting display problems with custom annotation tracks, refer to the troubleshooting section in the Creating custom annotation tracks section. Viewing genome data as text The Table Browser, a portal to the underlying open source MariaDB relational database driving the Genome Browser, displays genomic data as columns of text rather than as graphical tracks. For more information on using the Table Browser, see the section Getting started: on the Table Browser. Opening the Genome Browser from external gateways Several external gateways provide direct links into the Genome Browser. Examples include: Entrez Gene, AceView, Ensembl, SuperFamily, and GeneCards. Journal articles can also link to the browser and provide custom tracks. Be sure to use the assembly date appropriate to the provided coordinates when using data from a journal source. Tips for Use To facilitate your return to regions of interest within the Genome Browser, save the coordinate range or bookmark the page of displays that you plan to revisit or wish to share with others. It is usually best to work with the most recent assembly even though a full set of tracks might not yet be ready. Be aware that the coordinates of a given feature on an unfinished chromosome may change from one assembly to the next as gaps are filled, artifactual duplications are reduced, and strand orientations are corrected. The Genome Browser offers multiple tools that can correctly convert coordinates between different assembly releases. For more information on conversion tools, see the section Converting data between assemblies. To ensure uninterrupted browser services for your research during UCSC server maintenance and power outages, bookmark a mirror site that replicates the UCSC genome browser. Bear in mind that the Genome Browser cannot outperform the underlying quality of the draft genome. Assembly errors and sequence gaps may still occur well into the sequencing process due to regions that are intrinsically difficult to sequence. Artifactual duplications arise as unavoidable compromises during a build, causing misleading matches in genome coordinates found by alignment. Interpreting and fine-tuning the Genome Browser display The Genome Browser annotation tracks page displays a genome location specified through a Gateway search, a BLAT search, or an uploaded custom annotation track. There are five main features on this page: a set of navigation controls, a chromosome ideogram, the annotations tracks image, display configuration buttons, and a set of track display controls. The first time you open the Genome Browser, it will use the application default values to configure the annotation tracks display. By manipulating the navigation, configuration and display controls, you can customize the annotation tracks display to suit your needs. For a complete description of the annotation tracks available in all assembly versions supported by the Genome Browser, see the Annotation Track Descriptions section. The Genome Browser retains user preferences from session to session within the same web browser, although it never monitors or records user activities or submitted data. To restore the default settings, click the "Reset All User Settings" under the top blue Genome Browser menu. To return the display to the default set of tracks (but retain custom tracks and other configured Genome Browser settings), click the default tracks button on the Genome Browser page. Annotation track display conventions Annotation track descriptions: Each annotation track has an associated description page that contains a discussion of the track, the methods used to create the annotation, the data sources and credits for the track, and (in some cases) filter and configuration options to fine-tune the information displayed in the track. To view the description page, click on the mini-button to the left of a displayed track or on the label for the track in the Track Controls section. Annotation track details pages: When an annotation track is displayed in full, pack, or squish mode, each line item within the track has an associated details page that can be displayed by clicking on the item or its label. The information contained in the details page varies by annotation track, but may include basic position information about the item, related links to outside sites and databases, links to genomic alignments, or links to corresponding mRNA, genomic, and protein sequences. Gene prediction tracks: Coding exons are represented by blocks connected by horizontal lines representing introns. The 5' and 3' untranslated regions (UTRs) are displayed as thinner blocks on the leading and trailing ends of the aligning regions. In full display mode, arrowheads on the connecting intron lines indicate the direction of transcription. In situations where no intron is visible (e.g. single-exon genes, extremely zoomed-in displays), the arrowheads are displayed on the exon block itself. Pattern Space Layout (PSL) alignment tracks: Aligning regions (usually exons when the query is cDNA) are shown as black blocks. In dense display mode, the degree of darkness corresponds to the number of features aligning to the region or the degree of quality of the match. In pack or full display mode, the aligning regions are connected by lines representing gaps in the alignment (typically spliced-out introns), with arrowheads indicating the orientation of the alignment, pointing right if the query sequence was aligned to the forward strand of the genome and left if aligned to the reverse strand. Two parallel lines are drawn over double-sided alignment gaps, which skip over unalignable sequence in both target and query. For alignments of ESTs, the arrows may be reversed to show the apparent direction of transcription deduced from splice junction sequences. In situations where no gap lines are visible, the arrowheads are displayed on the block itself. To prevent display problems, the Genome Browser imposes an upper limit on the number of alignments that can be viewed simultaneously within the tracks image. When this limit is exceeded, the Browser displays the best several hundred alignments in a condensed display mode, then lists the number of undisplayed alignments in the last row of the track. In this situation, try zooming in to display more entries or to return the track to full display mode. For some PSL tracks, extra coloring to indicate mismatching bases and query-only gaps may be available. Chain tracks (2-species alignment): Chain tracks display boxes joined together by either single or double lines. The boxes represent aligning regions. Single lines indicate gaps that are largely due to a deletion in the genome of the first species or an insertion in the genome of the second species. Double lines represent more complex gaps that involve substantial sequence in both species. This may result from inversions, overlapping deletions, an abundance of local mutation, or an unsequenced gap in one species. In cases where there are multiple chains over a particular portion of the genome, chains with single-lined gaps are often due to processed pseudogenes, while chains with double-lined gaps are more often due to paralogs and unprocessed pseudogenes. In the fuller display modes, the individual feature names indicate the chromosome, strand, and location (in thousands) of the match for each matching alignment. Net tracks (2-species alignment): Boxes represent ungapped alignments, while lines represent gaps. Clicking on a box displays detailed information about the chain as a whole, while clicking on a line shows information on the gap. The detailed information is useful in determining the cause of the gap or, for lower level chains, the genomic rearrangement. Individual items in the display are categorized as one of four types (other than gap): - Top - The best, longest match. Displayed on level 1. - Syn - Lineups on the same chromosome as the gap in the level above it. - Inv - A lineup on the same chromosome as the gap above it, but in the opposite orientation. - NonSyn - A match to a chromosome different from the gap in the level above. Snake tracks: The snake alignment track (or snake track) shows the relationship between the chosen Browser genome (reference genome) and another genome (query genome). A snake is a way of viewing a set of pairwise gapless alignments that may overlap on both the reference and query genomes. Alignments are always represented as being on the positive strand of the reference species, but can be on either strand on the query sequence. In full display mode, a snake track can be decomposed into two drawing elements: segments (colored rectangles) and adjacencies (lines connecting the segments). Segments represent subsequences of the target genome aligned to the given portion of the reference genome. Adjacencies represent the covalent bonds between the aligned subsequences of the target genome. Red tick-marks within segments represent substitutions with respect to the reference, shown in windows of the reference of (by default) up to 50 Kb. Zoomed in to the base level, these substitutions are labeled with the non-reference base. An insertion in the reference relative to the query creates a gap between abutting segment sides that is connected by an adjacency. An insertion in the query relative to the reference is represented by an orange tick-mark that splits a segment at the location the extra bases would be inserted. Simultaneous independent insertions in both query and reference look like an insertion in the reference relative to the target, except that the corresponding adjacency connecting the two segments is colored orange. More complex structural rearrangements create adjacencies that connect the sides of non-abutting segments in a natural fashion. Pack mode can be used to display a larger number of snake tracks in the limited vertical browser. This mode eliminates the adjacencies from the display and forces the segments onto as few rows as possible, given the constraint of still showing duplications in the query sequence. Dense mode further eliminates these duplications so that each snake track is compactly represented along just one row. Wiggle tracks: These tracks plot a continuous function along a chromosome. Data is displayed in windows of a set number of base pairs in width. The score for each window displays as "mountain ranges" The display characteristics vary among the tracks in this group. See the individual track descriptions for more information on interpreting the display. If the peak is taller or shorter than what can be shown in the display, it is clipped and colored magenta. Annotation track display modes Each annotation track within the window may have up to five display modes: - Hide: the track is not displayed at all. To hide all the annotation tracks, click the hide all button. This mode is useful for restricting the display to only those tracks in which you are interested. For example, someone who is not interested in SNPs or mouse synteny may want to hide these tracks to reduce track clutter and improve speed. There are a few annotation tracks that pertain only to one specific chromosome, e.g. Sanger22, Rosetta. In these cases, the track and its associated controller will be hidden automatically when the track window is not open to the relevant chromosome. - Dense: the track is displayed with all features collapsed into a single line. This mode is useful for reducing the amount of space used by a track when you don't need individual line item details or when you just want to get an overall view of an annotation. For example, by opening an entire chromosome and setting the RefSeq Genes track to dense, you can get a feel for the known gene density of the chromosome without displaying excessive detail. [] - Full: the track is displayed with each annotation feature on a separate line. It is recommended that you use this option sparingly, due to the large number of individual track items that may potentially align at the selected position. For example, hundreds of ESTs might align with a specified gene. When the number of lines within a requested track location exceeds 250, the track automatically defaults to a more tightly-packed display mode. In this situation, you can restore the track display to full mode by narrowing the chromosomal range displayed or by using a track filter to reduce the number of items displayed. On tracks that contain only hide, dense, and full modes, you can toggle between full and dense display modes by clicking on the track's center label. [] - Squish: the track is displayed with each annotation feature shown separately, but at 50% the height of full mode. Features are unlabeled, and more than one may be drawn on the same line. This mode is useful for reducing the amount of space used by a track when you want to view a large number of individual features and get an overall view of an annotation. It is particularly good for displaying tracks in which a large number of features align to a particular section of a chromosome, e.g. EST tracks. [] - Pack: the track is displayed with each annotation feature shown separately and labeled, but not necessarily displayed on a separate line. This mode is useful for reducing the amount of space used by a track when you want to view the large number of individual features allowed by squish mode, but need the labeling and display size provided by full mode. When the number of lines within the requested track location exceeds 250, the track automatically defaults to squish display mode. In this situation, you can restore the track display to pack mode by narrowing the chromosomal range displayed or by using a track filter to reduce the number of items displayed. To toggle between pack and full display modes, click on the track's center label. [] The track display controls are grouped into categories that reflect the type of data in the track, e.g., Gene Prediction Tracks, mRNA and EST tracks, etc. To change the display mode for a track, find the track's controller in the Track Controls section at the bottom of the Genome Browser page, select the desired mode from the control's display menu, and then click the refresh button. Alternatively, you can change the display mode by using the Genome Browser's right-click navigation feature, or can toggle between dense and full modes for a displayed track (or pack mode when available) by clicking on the optional center label for the track. Duplicating a track Tracks that are not inside of composite or supertracks can be duplicated to allow for independent track settings for a track. To duplicate a track, go to the track settings page for the track and there will be a link, "Duplicate track", next to the Display mode setting: [] Clicking the link will take you to the new track settings page for the duplicate track with the additional text, "(duplicate #1)". This number will increment by one for each additional copy of the track. After creating the copy, a "Remove duplicate" link will also appear on the track settings page for when you wish to remove the duplicated track. As an example, duplicating the GENCODE track on hg38 allows users to have two tracks, one in 'squish' mode and a second track in 'full' mode as a 'density graph'. [] [] Each copy of the track will have it's own independent settings to allow for multiple display views without having to revert back to an alternate view for the dataset. [] Changing the display mode for a group of tracks Track display modes may be set individually or as a group on the Genome Browser Track Configuration page. To access the configuration page, click the configure button on the annotation tracks page or the configure tracks and display button on the Gateway page. Exercise caution when using the show all buttons on track groups or assemblies that contain a large number tracks; this may seriously impact the display performance of the Genome Browser or cause your Internet browser to time out. Hiding the track display controls The entire set of track display controls at the bottom of the annotation tracks page may be hidden from view by checking the Show track controls under main graphic option in the Configure Image section of the Track Configuration page. Changing the display of a track by using filters and configuration options Some tracks have additional filter and configuration capabilities, e.g., EST tracks, mRNA tracks, NC160, etc. These options let the user modify the color or restrict the data displayed within an annotation track. Filters are useful for focusing attention on items relevant to the current task in tracks that contain large amounts of data. For example, to highlight ESTs expressed in the liver, set the EST track filter to display items in a different color when the associated tissue keyword is "liver" Configuration options let the user adjust the display to best show the data of interest. For example, the min vertical viewing range value on wiggle tracks can be used to establish a data threshold. By setting the min value to "50", only data values greater than 50 percent will display. To access filter and configuration options for a specific annotation track, open the track's description page by clicking the label for the track's control menu under the Track Controls section, the mini-button to the left of the displayed track, or the "Configure..." option from the Genome Browser's right-click popup menu. The filter and configration section is located at the top of the description page. In most instances, more information about the configuration options is available within the description text or through a special help link located in the configuration section. Filter and configuration settings are persistent from session to session on the same web browser. To return the Genome Browser display to the default set of tracks (but retain custom tracks and other configured Genome Browser settings), click the default tracks button on the Genome Browser tracks page. To remove all user configuration settings and custom tracks, and completely restore the defaults, click the "Reset All User Settings" under the top blue Genome Browser menu. Video tutorial on changing track display modes []       Visit our Video Page. Visit our YouTube channel for more videos. Zooming and scrolling the tracks display At times you may want to adjust the amount of flanking region displayed in the annotation tracks window or adjust the scale of the display. At a scale of 1 pixel per base pair, the window accurately displays the width of exons and introns, and indicates the direction of transcription (using arrowheads) for multi-exon features. At a grosser scale, certain features - such as thin exons - may disappear. Also, some exons may falsely appear to fall within RepeatMasker features at some scales. Click the zoom in and zoom out buttons at the top of the Genome Browser page to zoom in or out on the center of the annotation tracks window by 1.5, 3 or 10-fold. Alternatively, you can zoom in 3-fold on the display by clicking anywhere on the Base Position track. In this case, the zoom is centered on the coordinate of the mouse click. To view the base composition of the sequence underlying the current annotation track display, click the base button. Quickly zoom to a specific region of interest by using the browser's "drag-and-select" feature. To define the region you wish to zoom to, click and hold the mouse button on one edge of the desired zoom area in the Base Position track, drag the mouse right or left to highlight the selection area, then release the mouse button. A "drag-and-select" popup will appear. Click on the "Zoom In" button to zoom in on the selected region. To disable the drag-and-select popup, check the "Don't show this dialog again and always zoom" checkbox. To drag-and-select (zoom) on a part of the image other than the Base Position track, depress the shift key before clicking and dragging the mouse. Note that the Enable advanced javascript features option on the Track Configuration page must be toggled on to use this feature. To scroll (pan) the view of the entire tracks image horizontally, click on the image and drag the cursor to the left or right, then release the mouse button, to shift the displayed region in the corresponding direction. The view may be scrolled by up to one image width. To scroll the annotation tracks horizontally by set increments of 10%, 50%, or 95% of the displayed size (as given in base pairs), click the corresponding move arrow. It is also possible to scroll the left or right side of the tracks by a specified number of vertical gridlines while keeping the position of the opposite side fixed. To do this, click the appropriate move start or move end arrow, located under the annotation tracks window. For example, to keep the left-hand display coordinate fixed but increase the right-hand coordinate, you would click the right-hand move end arrow. To increase or decrease the gridline scroll interval, edit the value in the move start or move end text box. Highlighting a region The browser's "drag-and-select" pop-up menu provides options to add single or multiple vertical highlights to selected regions, as described below: Main features in drag-and-select menu: - Use shift+drag or click-drag to enable the "drag-and-select" dialog box menu. - In the menu, a checkbox controls behavior for drag-and-select; you can hide the menu and always zoom with shift+select. If selected, re-enable via 'View - Configure Browser' (keyboard shortcut: c then f). - A "color picker" option allows for easy color selection of each highlight; you can also create multiple highlights (each with various colors if desired). - Hold Shift+drag to show the menu (or click+drag). - Hold Alt+drag to add a highlight (without displaying the menu). - Hold Ctrl+drag (Windows) or Cmd+drag (Mac) to zoom (without displaying the menu). - To cancel, press Esc anytime or drag mouse outside image. - Highlight the current position with the keyboard shortcut "h then m." - Clear all highlights with View - Clear Highlights (or keyboard shortcut h then c), or simply right-click on a highlight to remove all highlights. In the genome browser, there are also options for right-clicking: - Remove highlighting - Zoom in to a highlighted region - Highlight a gene - right-click on the gene (e.g., SOD1) and select "Highlight SOD1" Changing the displayed track position To display a completely different position in the genome, enter the new query in the position/search text box, then click the jump button. For more information on valid entries for this text box, refer to the Getting started section. If a chromosome image (ideogram) is available above the track display, click anywhere on the chromosome to move to that position (the current window size will be maintained). Select a region of any size by clicking and dragging in the image. Finally, hold the "control" key while clicking on a chromosome band to select the entire band. Changing the order of the displayed tracks To vertically reposition a track in the annotation track window, click-and-hold the mouse button on the side label, then drag the highlighted track up or down within the image. Release the mouse button when the track is in the desired position. To move an entire group of associated tracks (such as all the displayed subtracks in a composite track), click-and-hold the gray mini-button to the left of the tracks, then drag. Viewing multiple regions To remove intronic or intergenic regions from the display or to view only custom specified regions, click the multi-region button under the track image. For human assemblies hg17 and later, you may also replace a section of the reference genome with an alternate haplotype chromosome in order to view annotations upstream and downstream of the sequence. For more information about the multi-region feature see the multi-region help page. Configuration panel [Configuration menu options] Changing the width of the annotation track window The first time the annotation track window is displayed, or after the Genome Browser has been reset, the size of the track window is set by default to the width that best fits your Internet browser window. If you horizontally resize the browser window, you can automatically adjust the annotation track image size to the new width by clicking the resize button under the track image. To manually override the default width, enter a new value in the image width text box on the Track Configuration page, then click the submit button. The maximum supported width is 5000 pixels. Changing the width of the label area to the left of the image The item labels (or track label, when viewed in dense mode) are displayed to the left of the annotation image. The width of this area is set to 17 characters by default. To change the width, edit the value in the label area width text box on the Track Configuration page, then click Submit. Changing the text size, font, in the annotation track image The annotation track image may be adjusted to display text in a range of fonts from AvantGarde, Courier, and Times. To change the size of the text, select an option from the text size pull-down menu on the Browser Configuration page, which you can find under the top blue "Genome Browser" menu by clicking Configure. Once you have made your selection, which can also include style click submit. The text size is set to "12" and "Helvetica" by default. Hiding the chromosome ideogram The chromosome ideogram, located just above the annotation tracks image, provides a graphical overview of the features on the selected chromosome, including its bands, the position of the centromere, and an indication of the region currently displayed in the annotation tracks image. To hide the ideogram, uncheck the Display chromosome ideogram above main graphic box on the Tracks Configuration page. Hiding the light blue vertical guidelines The light blue vertical guidelines on the annotation tracks image may be removed by unchecking the Show light blue vertical guidelines box on the Track Configuration page. Hiding the annotation track labels and description The track and element labels displayed above and to the left of the tracks in the annotation tracks image may be hidden from view by unchecking the Display track descriptions above each track and Display labels to the left of items in tracks boxes, respectively, on the Track Configuration page. Enabling next/previous item and exon navigation [Ex. white/gray arrows for next/previous exon/item] Gray arrows jump to the next item, while white arrows advance to the next exon. When the Next/previous item navigation configuration option is toggled on, on the Track Configuration page, gray double-headed arrows display in the Genome Browser tracks image on both sides of the track labels of gene, mRNA and EST tracks (or any standard tracks based on BED, PSL or genePred format). Clicking on the gray arrows shifts the image window toward that end of the chromosome so that the next item in the track is displayed. Similarly, the Next/previous exon navigation configuration option displays white double-headed arrows on the end of any item that extends off the edge of the current image. Clicking on one of the white arrows shifts the image window to the next exon in the indicated direction, unless the image window interrupts an exon, in which case the window shifts to the edge of the current exon. If the image window happens to be within a 5' or 3' UTR, then clicking the arrows shifts the image window towards the start or end of the next coding region, not the end of the exon. Using the right-click navigation feature Several of the common display and navigation operations offered on the Genome Browser tracks page may be quickly accessed by right-clicking on a feature on the tracks image and selecting an option from the displayed popup menu. [Example right-click to highlight a gene] Depending on context, the right-click feature allows the user to: - change the track display mode - zoom in or out to the exact position coordinates of the feature - highlight the feature - open the "Get DNA for..." link for the feature's coordinates - display details about the feature - open a popup window to configure the track's display of the feature - display the entire tracks image in a separate window for inclusion in spreadsheets or other documents. The Genome Browser PDF option under the View menu can also be used to generate a high-quality annotation tracks image suitable for printing. To use the right-click feature, make sure your internet browser allows the display of popup windows from genome.ucsc.edu. When enabled, the right-click navigation feature replaces the default contextual popup menu typically displayed by the Internet browser when a user right-clicks on the tracks image. A few combinations of the Mozilla Firefox browser on Mac OS do not support the right-click menu functionality using secondary click; in these instances, ctrl+left-click must be used to display the menu. Printing a copy of the annotation track window The Genome Browser provides a mechanism for saving a copy of the currently displayed annotation tracks image to a PDF file that can be printed or edited by drawing programs such as Adobe Illustrator or Inkscape. This is useful for generating figures intended for publication. To print or save the image to a file: 1. In the blue navigation bar at the top of the screen, from the "View" menu, click the "PDF" link. 2. Click one of the PDF links. NOTE: If you have configured your browser image to use one of the larger font sizes, the text in the resulting screen shot may not display correctly. If you encounter this problem, reduce the Genome Browser font size using the Configuration utility, then repeat the save/print process. Using BLAT alignments BLAT (BLAST-Like Alignment Tool) is a very fast sequence alignment tool similar to BLAST. For more information on BLAT's internal scoring schemes and its overall n-mer alignment seed strategy, refer to W. James Kent (2002) BLAT - The BLAST-Like Alignment Tool, Genome Res 12:4 656-664. On DNA queries, BLAT is designed to quickly find sequences with 95% or greater similarity of length 25 bases or more. It may miss genomic alignments that are more divergent or shorter than these minimums, although it will find perfect sequence matches of 32 bases and sometimes as few as 22 bases. The tool is capable of aligning sequences that contain large introns. On protein queries, BLAT rapidly locates genomic sequences with 80% or greater similarity of length 20 amino acids or more. In general, gene family members that arose within the last 350 million years can generally be detected. More divergent sequences can be aligned to the human genome by using NCBI's BLAST and psi-BLAST, then using BLAT to align the resulting match onto the UCSC genome assembly. In practice DNA Blat works well on primates, and protein Blat works well on land vertebrates. Some common uses of BLAT include: - finding the genomic coordinates of mRNA or protein within a given assembly - determining the exon structure of a gene - displaying a coding region within a full-length gene - isolating an EST of special interest as its own track - searching for gene family members - finding human homologs of a query from another species - finding homologs of a query in all species hosted on the UCSC Genome Browser Making a BLAT query To locate a nucleotide or protein within a genome using BLAT: 1. Open the BLAT Search Genome page by clicking on the "Tools" pulldown in the top blue menu bar of the Genome Browser. 2. Blat a single genome – Select the genome, assembly, query type, output sort order, and output type. To order the search results based on the closeness of the sequence match, choose one of the score options in the Sort output menu. The score is determined by the number of matches vs. mismatches in the final alignment of the query to the genome. Blat ALL genomes – The "Search ALL" checkbox above the Genome drop-down list allows you to search the genomes of the default assemblies for all of our organisms. This shows you an ordered list of the default assemblies having the greatest similarity with your query sequence. Additionally, the Search ALL feature searches any attached hubs' blat servers, meaning you can search your user-generated assembly hubs. 3. If the sequence to be uploaded is in an unformatted plain text file, enter the file name in the Upload sequence text box, then click the submit file button. Otherwise, paste the sequence or fasta-formatted list into the large edit box, and then click the submit button. Input sequence can be obtained from the Genome Browser as well as from a custom annotation track. Header lines may be included in the input text if they are preceded by > and contain unique names. Multiple sequences may be submitted at the same time if they are of the same type and are preceded by unique header lines. Numbers, spaces, and extraneous characters are ignored: >sequence_1 ATGCAGAGCAAGGTGCTGCTGGCCGTCGCCCTGTGGCTCTGCGTGGAGAC CCGGGCCGCCTCTGTGGGTTTGCCTAGTGTTTCTCTTGATCTGCCCAGGC >sequence_2 ATGTTGTTTACCGTAAGCTGTAGTAAAATGAGCTCGATTGTTGACAGAGA TGACAGTAGTATTTTTGATGGGTTGGTGGAAGAAGATGACAAGGACAAAG >sequence_3 ATGCTGCGAACAGAGAGCTGCCGCCCCAGGTCGCCCGCCGGACAGGTGGC CGCGGCGTCCCCGCTCCTGCTGCTGCTGCTGCTGCTCGCCTGGTGCGCGG BLAT limitations DNA input sequences are limited to a maximum length of 25,000 bases. Protein or translated input sequences must not exceed 10,000 letters. As many as 25 multiple sequences may be submitted at the same time. The maximum combined length of DNA input for multiple sequence submissions is 50,000 bases (with a 25,000 base limit per individual sequence). For protein or translated input, the maximum combined input length is 25,000 letters (with a 5000 letter limit per individual sequence). NOTE: Program-driven BLAT use is limited to a maximum of one hit every 15 seconds and no more than 5000 hits per day. BLAT query search results If a query returns successfully, BLAT will display a flat database file that summarizes the alignments found. A BLAT query often generates multiple hits. This can happen when the genome contains multiple copies of a sequence, paralogs, pseudogenes, statistical coincidences, artifactual assembly duplications, or when the query itself contains repeats or common retrotransposons. When too many hits occur, try resubmitting the query sequence after filtering in slow mode with RepeatMasker. Items in the search results list are ordered by the criteria specified in the Sort output menu. Each line item provides links to view the details of the sequence alignment or to open the corresponding view in the Genome Browser. The details link gives the letter-by-letter alignment of the sequence to the genome. It is recommended that you first examine the details of the alignment for match quality before viewing the sequence in the Genome Browser. When several nearby BLAT matches occur on a single chromosome, a simple trick can be used to quickly adjust the Genome Browser track window to display all of them: open the Genome Browser with the match that has the lowest chromosome start coordinate, paste in the highest chromosome end coordinate from the list of matches, then click the jump button. Creating a custom annotation track from BLAT output To make a custom track directly from BLAT, select the PSL format output option. The resulting PSL track can be uploaded into the Genome Browser by pasting the data into the data text box on the Genome Browser Add Custom Tracks page, accessed via the "add custom tracks" button on the Browser gateway and annotation tracks pages. See the Creating custom annotation tracks section for more information. Using BLAT for large batch jobs or commercial use For large batch jobs or internal parameter changes, it is best to install command line BLAT on your own Linux server. Sources and executables are free for academic, personal, and non-profit purposes. BLAT source may be downloaded from https://genome-test.gi.ucsc.edu/~kent/src/ (look for the blatSrc*.zip file with the most recent date). For BLAT executables, go to http://genome-test.soe.ucsc.edu/~kent/exe/; binaries are sorted by platform. Non-exclusive commercial licenses are available from the Kent Informatics website. BLAT documentation For more information on the BLAT suite of programs, see the BLAT Program Specifications and the Blat section of the Genome Browser FAQ. Annotation track descriptions Detailed information about an individual annotation track, including display characteristics, configuration information, and associated database tables, may be obtained from the track description page accessed by clicking the mini-button to the left of the displayed track in the Genome Browser, or by selecting the "Open details..." or "Show details..." option from the Genome Browser's right-click menu. Click the "View data format description" link on the track description page to display additional information about the primary database table underlying the track. Data format information may also be accessed via the "data format description" button in the Table Browser. For more information on configuring and using the tracks displayed in the Genome Browser track window, see the section Interpreting and Fine-tuning the Genome Browser display. Tips for viewing annotation track data - To display a description page with more information about the track, click on the mini-button to the left of a track. - To display a details page with additional information about a specific line item within a track in full display mode, click on the item or its label. - A track does not appear in the browser if its display mode is set to hide. To restrict the browser's display to only those tracks in which you're interested, set the display mode of the unwanted tracks to hide. - A track set to full display mode will default to a more tightly packed display mode if the total number of lines in the track exceeds 250. - To quickly toggle between full and dense or pack display modes, click on the track's center label. - Only the most recent assemblies are fully active. The data for older assemblies may be available on our Downloads page. - Not all tracks appear in all assemblies. Only a basic set of tracks appears initially in a new assembly. - Track data can be viewed as text tables using the Table Browser. - Credit goes to many individuals and institutions for generously contributing the tracks. For specific information about the contributors of a given track, look at the Credits section on a track's description page. Getting started on the Table Browser The Table Browser provides text-based access to the genome assemblies and annotation data stored in the Genome Browser database. As a flexible alternative to the graphical-based Genome Browser, this tool offers an enhanced level of query support that includes restrictions based on field values, free-form SQL queries, and combined queries on multiple tables. Output can be filtered to restrict the fields and lines returned, and may be organized into one of several formats, including a simple tab-delimited file that can be loaded into a spreadsheet or database as well as advanced formats that may be uploaded into the Genome Browser as custom annotation tracks. The Table Browser provides a convenient alternative to downloading and manipulating the entire genome and its massive data tracks. (See the Downloading Genome Data section.) For information on using the Table Browser features, refer to the Table Browser User Guide. Getting started using Sessions The Sessions tool allows users to configure their browsers with specific track combinations, including custom tracks, and save the configuration options. Multiple sessions may be saved for future reference, for comparison of scenarios or for sharing with colleagues. Saved sessions will not be expired, however we still recommend that you keep local back-ups of your session contents and any associated custom tracks. User-generated tracks can be saved within sessions. This tool may be accessed by clicking the "My Data" pulldown in the top blue navigation bar in any assembly and then selecting Sessions. To ensure privacy and security, you must create an account and/or log in to use the Session tool. Individual sessions may be designated by the user as either "shared" or "non-shared" to protect the privacy of confidential data. To avoid having a new shared session from someone else override existing Genome Browser settings, users are encouraged to open a new web-browser instance or to save existing settings in a session before loading a new shared session. For more detailed information on using the Session tool, see the Sessions User Guide. Getting started on Genome Graphs The Genome Graphs tool can be used to display genome-wide data sets such as the results of genome-wide SNP association studies, linkage studies, and homozygosity mapping. This tool is not pre-loaded with any sample data; instead, you can upload your own data for display by the tool. Once you have uploaded your data, you can view it in a variety of ways. You can view multiple sets of genome-wide data simultaneously either as superimposed graphs or side-by-side graphs. Once you see an area of interest in the Genome Graphs view, you can click on it to go directly to the Genome Browser at that position. You can also set a significance threshold for your data and view only regions or gene sets that meet that threshold. For information on using the Genome Graphs features, refer to the Genome Graphs User Guide. Using the VisiGene Image Browser VisiGene is a browser for viewing in situ images. It enables the user to examine cell-by-cell as well as tissue-by-tissue expression patterns. The browser serves as a virtual microscope, allowing users to retrieve images that meet specific search criteria, then interactively zoom and scroll across the collection. To start the VisiGene browser, click the VisiGene link in the left-hand sidebar menu on the Genome Browser home page. Images Available The following image collections are currently available for browsing: - High-quality high-resolution images of eight-week-old male mouse sagittal brain slices with reverse-complemented mRNA hybridization probes from the Allen Brain Atlas, courtesy of the Allen Institute for Brain Science - Mouse in situ images from the Jackson Lab Gene Expression Database (GXD) at MGI - Transcription factors in mouse embryos from the Mahoney Center for Neuro-Oncology - Mouse head and brain in situ images from NCBI's Gene Expression Nervous System Atlas (GENSAT) database - Xenopus laevis in situ images from the National Institute for Basic Biology (NIBB) XDB project Searching the Image Database The image database may be searched by gene symbols, authors, years of publication, body parts, GenBank or UniProtKB accessions, organisms, Theiler stages (mice), and Nieuwkoop/Faber stages (frogs). The search returns only those images that match all the specified criteria. For a list of sample search strings, see the VisiGene Gateway page. The wildcard characters * and ? are supported for gene name searches. For example, to view the images of all genes in the Hox A cluster, search for hoxa*. When searching on author names that include initials, use the format Smith AJ. Image Navigation Following a successful search, VisiGene displays a list of thumbnails of images matching the search criteria in the lefthand pane of the browser. By default, the image corresponding to the first thumbnail in the list is displayed in the main image pane. If more than 25 images meet the search criteria, links at the bottom of the thumbnail pane allow the user to toggle among pages of search results. To display a different image in the main browser pane, click the thumbnail of the image you wish to view. By default, an image is displayed at a resolution that provides optimal viewing of the overall image. This size varies among images. The image may be zoomed in or out, sized to match the resolution of the original image or best fit the image display window, and moved or scrolled in any direction to focus on areas of interest. The original full-sized image may also be downloaded. Zooming in: To enlarge the image by 2X, click the Zoom in button above the image or click on the image using the left mouse button. Alternatively, the + key may be used to zoom in when the main image pane is the active window. Zooming out: To reduce the image by 2X, click the Zoom out button above the image or click on the image using the right mouse button. Alternatively, the - key may be used to zoom out when the main image pane is the active window. Sizing to full resolution: Click the Zoom full button above the image to resize the image such that each pixel on the screen corresponds to a pixel in the digitized image. Sizing to best fit: Click the Zoom fit button above the image to zoom the image to the size that best fits the main image pane. Moving the image: To move the image viewing area in any direction, click and drag the image using the mouse. Alternatively, the following keyboard shortcuts may be used after clicking on the image: - Scroll left in the image: Left-arrow key or Home key - Scroll right in the image: Right-arrow key or End key - Scroll up in the image: Up-arrow key or PgUp key - Scroll down in the image: Down-arrow key or PgDn key Downloading the original full-sized image: Most images may be viewed in their original full-sized format by clicking the "download" link at the bottom of the image caption. NOTE: due to the large size of some images, this action may take a long time and could potentially exceed the capabilities of some Internet browsers. If you have an image set you would like to contribute for display in the VisiGene Browser, contact Jim Kent. DNA text formatting The Genome Browser provides a feature to configure the retrieval, formatting, and coloring of the text used to depict the DNA sequence underlying the features in the displayed annotation tracks window. Retrieval options allow the user to add a padding of extra bases to the upstream or downstream end of the sequence. Formatting options range from simply displaying exons in upper case to elaborately marking up a sequence according to multiple track data. The DNA sequence covered by various tracks can be highlighted by case, underlining, bold or italic fonts, and color. The DNA display configuration feature can be useful to highlight features within a genomic sequence, point out overlaps between two types of features (for example, known genes vs. gene predictions), or mask out unwanted features. Using the DNA text formatting feature To access the feature, click on the "View" pulldown on the top blue menu bar on the Genome Browser page and select "DNA", or select the "Get DNA..." option from the Genome Browser's right-click menu depending on context. "The Get DNA in Window" page that appears contains sections for configuring the retrieval and output format. To display extra bases upstream of the 5' end of your sequence or downstream of the 3' end of the sequence, enter the number of bases in the corresponding text box. This option is useful in looking for regulatory regions. The Sequence Formatting section lists several options for adjusting the case of all or part of the DNA sequence. To choose one of these formats, click the corresponding option button, then click the get DNA button. To access a table of extended formatting options, click the Extended case/color options button. The Extended DNA Case/Color page presents a table with many more format options. The page provides instructions for using the formatting table, as well as examples of its use. The list of tracks in the Track Name column is automatically generated from the list of tracks available on the current genome. Tips for Use A few caveats mentioned on the Extended DNA Case/Color page bear repeating. Keep the formatting simple at first: it is easy to make a display that is pretty to look at but is also completely cryptic. Also, be careful when requesting complex formatting for a large chromosomal region: when all the HTML tags have been added to the output page, the file size may exceed the size limits that your Internet browser, clipboard, and other software can safely display. The maximum size of genome that can be formatted by the tool is approximately 10 Mbp. Converting data between assemblies Coordinates of features frequently change from one assembly to the next as gaps are closed, strand orientations are corrected, and duplications are reduced. Occasionally, a chunk of sequence may be moved to an entirely different chromosome as the map is refined. There are three different methods available for migrating data from one assembly to another: BLAT alignment, coordinate conversion, and coordinate lifting. The BLAT alignment tool is described in the section Using BLAT alignments. Coordinate conversion The Genome Browser Convert utility is useful for locating the position of a feature of interest in a different release of the same genome or (in some cases) in a genome assembly of another species. During the conversion process, portions of the genome in the coordinate range of the original assembly are aligned to the new assembly while preserving their order and orientation. In general, it is easier to achieve successful conversions with shorter sequences. When coordinate conversion is available for an assembly, click on the "View" pulldown on the top blue menu bar on the Genome Browser page and select the "In Other Genomes (Convert)" link. You will be presented with a list of the genome/assembly conversion options available for the current assembly. Select the genome and assembly to which you'd like to convert the coordinates, then click the Submit button. If the conversion is successful, the browser will return a list of regions in the new assembly, along with the percent of bases and span covered by that region. Click on a region to display it in the browser. If the conversion is unsuccessful, the utility returns a failure message. Lifting coordinates The liftOver tool is useful if you wish to convert a large number of coordinate ranges between assemblies. This tool is available in both web-based and command line forms, and supports forward/reverse conversions as well as conversions between species. You can use the BED format (e.g. "chr4 100000 100001", 0-based) or directly paste text from the position box ("chr4:100,001-100,001", 1-based). See our Coordinate Counting blog post for a discussion of the difference. If the coordinates do not cover a single base pair e.g. "chr4 100000 100000" (BED) or "chr4:100,001-100,000" (text), this tool automatically extends them to at least one base pair. Note: It is not recommeneded to use LiftOver to convert SNPs between assemblies, and more information about how to convert SNPs between assemblies can be found on the following FAQ entry. Web-based coordinate lifting To access the graphical version of the liftOver tool, click on "Tools" pulldown in the top blue menu bar of the Genome Browser, then select LiftOver from the menu. Note that the web tool has an input file size limit of 500Mb, larger files will require using the command-line version. To convert one or more coordinate ranges using the default conversion settings: 1. Select the genome and assembly from which the ranges were taken ("Original"), as well as the genome and assembly to which the coordinates should be converted ("New"). 2. Select the Data Format option: Browser Extensible Data format (BED) or position (coordinates of the form chrN:start-end). 3. Enter coordinate ranges in the selected data format into the large text box, one per line. 4. Click Submit. Alternatively, you may load the coordinate ranges from an existing data file by entering the file name in the upload box at the bottom of the screen, then clicking the Submit File button. LiftOver parameters The default parameters for LiftOver are recommended for general use of the LiftOver tool. However, you may want to customize settings if you have several very large regions to convert. Although the LiftOver program accepts position ranges as input, some settings are only available when using a BED line with standard columns. The following describes each setting and the input requirements to use each setting. - Position Coordinates or BED with 3 standard columns: Minimum ratio of bases that must remap It may be beneficial to reduce this value when working with poor-quality assemblies or cross-species conversions. For large-area conversions under these conditions, a value as low as 0.01 may be used. [default: 0.95] - BED with 4 standard columns or greater: Allow multiple output regions By default, liftOver does not return a match if the region is split in the new assembly. If LiftOver can map a region more than once, and this setting isn't on, there will be no output for the region. Consider checking this option for conversions involving high-quality data within the same species. This option should not be used when mapping large regions, doing cross-species conversions, or using fragmented poor-quality assemblies. [default: off] Minimum chain size in target Only applicable when the "Allow multiple output regions" option is checked. Increasing the minimum chain size in the target may reduce fragmentation. A low-complexity or repetitive region may have lots of little matches across the genome that don't necessarily imply the regions had a common ancestor or are orthologous. For example, when doing large-scale conversions, we have found that a setting of 4,000 worked well for this option. [default: 0] Minimum hit size in query Only applicable when the "Allow multiple output regions" option is checked. Consider using this filter to remove small targets below a certain length that may be introduced by a repeat/transposon within a longer input region. [default: 0] - BED with 12 standard columns: Min ratio of alignment blocks/exons that must map The minimum ratio of the number of exons (not their bases) covered by the alignment. Transcripts lower than this will not be output at all. For example, with a value of 3, there must be 3 exons from the transcript covered in the alignment to produce output. With a value of 0.5, only half of an exon must be covered in the alignment. [default: 1] If thickStart/thickEnd is not mapped, use the closest mapped base Recommended if adjusting the "minimum ratio of alignment blocks/exons that must map". By default, if an exon (defined by thickStart to thickEnd in the BED) is not alignable at all, it will be skipped in the alignment. If this option is checked, the exon is lifted to the closest alignable base. [default: off] Command-line coordinate lifting The command-line version of liftOver offers the increased flexibility and performance gained by running the tool on your local server. See an example of running the liftOver tool on the command line. This utility requires access to a Linux platform. The executable file may be downloaded here. Command-line liftOver requires a UCSC-generated over.chain file as input. Pre-generated files for a given assembly can be accessed from the assembly's "LiftOver files" link on the Downloads page. If the desired conversion file is not listed, send a request to the genome mailing list and we may be able to generate one for you. For use of the command-line version of LiftOver, we require all for-profit businesses or commercial companies to purchase a license to support our small team. Downloading genome data Most of the underlying tables containing the genomic sequence and annotation data displayed in the Genome Browser can be downloaded. All of the tables are freely usable for any purpose except as indicated in the README.txt file in the download directories. This data was contributed by many researchers, as listed on the Genome Browser Credits page. Please acknowledge the contributor(s) of the data you use. Downloading the data Genome data can be downloaded in different ways using our North American and European download servers, hgdownload, hgdownload2, and hgdownload-euro. Via rsync: The UCSC Genome Browser hgdownload server contains download directories for all genome versions currently accessible in the Genome Browser. Either of the following rsync commands can quickly and efficiently download large files to your current directory (./). To download an entire directory (note the trailing slash), you would use an expression such as: For more information please click here. Via ftp: The UCSC Genome Browser FTP server contains download directories for all genome versions currently accessible in the Genome Browser. The FTP URLs ftp://hgdownload.soe.ucsc.edu/goldenPath/, ftp://hgdownload2.soe.ucsc.edu/goldenPath/, or ftp://hgdownload-euro.soe.ucsc.edu/goldenPath/ will take you to a directory that contains the genome download directories. This download method is not recommended if you plan to download a large file or multiple files from a single directory compared to rsync (see above). You can, however, use the mget command to download multiple files: mget filename1 filename2, or mget -a (to download all the files in the directory). Via the Downloads link: Click the Downloads link on the left side bar on the UCSC Genome Browser home page to display a list of all database directories available for download. If the data you wish to download pre-dates the assembly versions listed, look for the data on our Downloads page. Via the REST API: The REST API can be used to query both annotation and sequence data from any UCSC genome assembly or hub. Types of data available There may be several download directories associated with each version of a genome assembly: the full data set (bigZips), the full data set by chromosome (chromosome), the annotation database tables (database), and one or more sets of comparative cross-species alignments. BigZips contains the entire draft of the genome in chromosome and/or contig form. Depending on the genome, this directory may contain some or all of the following files: - chromAgp.zip: Description of how the assembly was generated, unpacking to one file per chromosome. - chromFa.zip: The assembly sequence chromosomes, in one file per chromosome. Repeats from RepeatMasker and Tandem Repeats Finder are shown in lower case; non-repeating sequence is in upper case. The main assembly is contained in the chrN.fa files, where chrN is the name of the chromosome. The chrN_random.fa files contain clones that are not yet finished or cannot be placed with certainty at a specific place on the chromosome. In some cases, including the human HLA region on chromosome 6, the chrN_random.fa files also contain haplotypes that differ from the main assembly. - chromFaMasked.zip: The assembly sequence chromosomes, in one file per chromosome. Repeats are masked by capital Ns; non-repeating sequence is shown in upper case. - chromOut.zip: RepeatMasker .out file for chromosomes, generated by RepeatMasker at the -s sensitive setting. - chromTrf.zip: Tandem Repeats Finder locations, filtered to keep repeats with period less than or equal to 12, translated into one .bed file per chromosome. - contigAgp.zip: Description of how the assembly was generated from fragments at a contig layout level. - contigFa.zip: The assembly sequence contigs, in one file per contig. All contigs are in forward orientation relative to the chromosome. In some cases, this means that contigs will be reversed relative to their orientation in the NCBI assembly. Repeats are shown in lower case; non-repeating sequence is shown in upper case. - contigFaMasked.zip: The assembly sequence contigs, in one file per contig. Repeats are masked by capital Ns; non-repeating sequence is shown in upper case. - contigOut.zip: RepeatMasker .out file for contigs, generated by RepeatMasker at the -s sensitive setting. - contigTrf.zip: Tandem Repeats Finder locations, filtered to keep repeats with period less than or equal to 12, and translated into one .bed file per contig. - database.zip: The Genome Browser database as tab-delimited files and associated MariaDB (MySQL) table-creation tiles (eliminated in later assemblies due to size restrictions). - est.fa.zip: Sequences of all GenBank ESTs for the selected species. - liftAll.zip: The offsets of contigs within chromosomes. - mrna.zip: mRNAs in GenBank from the selected species. - refmrna.zip: RefSeq mRNAs from the selected species. - upstream1000.zip: Sequences 1000 bases upstream of annotated transcription start of RefSeq genes. This includes only cases where the transcription start is annotated separately from the coding region start. - upstream2000.zip: Same as upstream1000, but with 2000 bases. - upstream5000.zip: Same as upstream1000, but with 5000 bases. - xenoMrna.zip: All GenBank mRNAs from species other than that of the selected one. Chromosomes contains the assembled sequence for the genome in separate files for each chromosome in a zipped fasta format. The main assembly can be found in the chrN.fa files, where N is the name of the chromosome. The chrN_random.fa files contain clones that are not yet finished or cannot be placed with certainty at a specific place on the chromosome. In some cases, the chrN_random.fa files also contain haplotypes that differ from the main assembly. Database contains all of the positional and non-positional tables in the genome annotation database. Each table is represented by 2 files: - .sql file: the SQL commands used to create the table. - .txt.gz file: the MariaDB database table data in tab-delimited format and compressed with gzip. Schema descriptions for all tables in the genome annotation database may be viewed by using the "data format description" button in the Table Browser. Cross-species alignments directories, such as the vsMm4 and humorMm3Rn3 directories in the hg16 assembly, contain pairwise and multiple species alignments and filtered alignment files used to produce cross-species annotations. For more information, refer to the READMEs in these directories and the description of the Multiple Alignment Format (MAF). Getting started on Track Hubs Track hubs are web-accessible directories of genomic data that can be viewed on the UCSC Genome Browser alongside native annotation tracks. Hubs are a useful tool for visualizing a large number of genome-wide data sets. The Track Hub utility allows efficient access to data sets from around the world through the familiar Genome Browser interface. Browser users can display tracks from any public track hub that has been registered with UCSC. We offer guidelines for those who want to make a hub a public track hub. Additionally, users can import data from unlisted hubs or can set up, display, and share their own track hubs. For information on using the Track Hub features, refer to the Genome Browser Track Hub User Guide. For specific information on configuring your trackDb.txt file, refer to the Track Database Definition Document. See also the Basic Hub Quick Start Guide, Quick Start Guide to Organizing Track Hubs into Groupings, Track hub settings blog post, Quick Start Guide to Assembly Hubs, Quick Start Guide to Searchable Track Hubs, and Loading Track Hubs and Assembly Hubs with the URL. Track Hubs for GenArk assembly hub To construct a track hub that will display on a GenArk Assembly hub, specify the GenArk assembly name in the genome statement in your hub.txt file as described below. For example, a reference such as: genome GCA_021951015.1 will direct your track hub to display on the human Feb. 2022 - GCA_021951015.1 - HG002.mat.cur.20211005 genome assembly. To share your track hub with your audience of interest, when you publish the URL to your track hub, that genome reference in your track hub.txt file will cause that associated assembly hub to display in the genome browser with your track hub annotations on that genome browser. There is no need to otherwise reference the assembly hub, it will automatically attach itself. A complete list of all available GenArk assemblies available can be seen in the text file UCSC_GI.assemblyHubList.txt. Track Hubs in a single file Historically, a hub needed a set of text files to specify properties for a track hub and each of the data tracks within the hub. The track hub settings were stored in a three file structure: hub.txt, genomes.txt, and trackDb.txt myHub/ - directory containing track hub files * hub.txt - a short description of hub properties * genomes.txt - list of genome assemblies included in the hub data * hg19/ - directory of data for the hg19 (GRCh37) human assembly ** trackDb.txt - display properties for tracks in this directory Now, there is a trackDb option to have your entire track hub inside of one file, useOneFile on. Adding the useOneFile on line to the hub.txt section of the file allows the contents of all three files to be referenced inside of one file. We recommend the single hub.txt file's contents be in the following order: hub.txt, genomes.txt, then trackDb.txt. hub myExampleHub shortLabel Example Hub longLabel Example Hub for useOneFile option useOneFile on email genome-www@soe.ucsc.edu genome hg19 track vcfExample shortLabel VCF example longLabel VCF: 1000 Genomes phase 1 interim SNVs visibility pack type vcfTabix maxWindowToDraw 200000 bigDataUrl http://genome.ucsc.edu/goldenPath/help/examples/vcfExample.vcf.gz track bamExample shortLabel BAM example longLabel Bam: 1000 Genomes read alignments (individual NA12878) visibility squish type bam pairSearchRange 10000 bamColorMode grey maxWindowToDraw 200000 bigDataUrl http://genome.ucsc.edu/goldenPath/help/examples/bamExample.bam The hub above can be accessed using the following URL to the single file, singleFileHubExample.txt. Limitations Unfortunately, the useOneFile on setting limits the hub to one assembly. The useOneFile on setting works by having the genomes.txt file point to only one trackDb.txt file, which is the file itself. Although this method limits you to one genome assembly, using the setting grants the advantage of not embedding file names in the hub architecture. So a simple name change to a hub file will no longer require editing the contents inside the hub.txt and genomes.txt files. Converting an existing Track Hub to one file Converting an existing track hub to use the new setting does not require much editing. Simply concatenate the three files into one, with the contents of hub.txt, genomes.txt, and trackDb.txt, in that order. Using the URL to the single file on the Connected Hubs page will allow you to view your track hub. 
/goldenPath/help/hgWiggleTrackHelp.html:Genome_Browser_Wiggle_Tracks	Configuring graph-based tracks Genome Browser graph-based tracks may be configured in a variety of ways to highlight different aspects of the displayed information. Some track types, such as BAM tracks and certain gene tracks, offer dynamic calculation of items in the current window to display a density graph where the height is proportional to the number of items mapped to each genomic position. The following information explains each of the available configuration options. Display settings [] Type of graph: The default "Bar" setting depicts the graph data using color-filled bars. To view the data as a series of points or lines, select the "Points" setting. Track height (boxed in blue): To change the default display height of the graph in pixels, type in a value from within the indicated range. Data view scaling (boxed in red): The default "Use vertical viewing range setting" option displays the data using the parameters specified in the Vertical viewing range setting. To configure the graph to automatically scale to a range defined by the minimum and maximum data points in the current view, select the "Auto-scale to data view" option. To keep the y=0 value in view at all times when Auto-scale is selected, set "Always include zero" to "ON". - When viewing signal tracks within a composite group use "group auto-scale" to enable having all tracks scaled against the one track in the group that has the highest maximum data points in the current view. For example, below is a side-by-side image of two views of the same data from a selection of cell lines within a composite of related RNA-seq experiments. On the left is the original "auto-scale to data view" setting, where each track is auto-scaled to appear at each track's highest value. And on the right is the new "group auto-scale" setting for the same RNA-seq data where all tracks are scaled against the one track in the region that has the highest value (67215 for IMR9 cell TAP + 1). [Auto-scale comparison] Click this link to interact with the original "auto-scale to data view" on the left, and to contrast the results with the new "group auto-scale" click this link. - Note that the "group auto-scale" setting is meant to be turned on and off at the composite group track set level, but you can toggle individual tracks. For instance, click the second "group auto-scale" session link that has all tracks adjusted to the highest value (67215 for IMR9 cell TAP + 1). If you right-click that individual highest track and instead of configuring the track set, select to configure only the subtrack IMR9 cell TAP + 1 from "group auto-scale" to "auto-scale to data view" and select OK and then refresh your browser window (or click one of the "refresh" buttons on the screen) you will see all the tracks adjust to the next-highest track within all tracks still tagged to display "group auto-scale" (39320 for IMR9 cell + 1). Vertical viewing range (boxed in red): The min and max values specify the vertical portion of the graph that is displayed (default range is 30-70). These numbers can be used to set data threshold indicators. For example, to display only those GC values greater than 50 percent, set the min value to "50". Transform function: Transforms the data points by the function selected in the drop-down menu. Usually the default setting is "None". Windowing function: When a view is too large to show individual data values, the values must be combined to produce a plot point. This option specifies the combining function to be used (default is "Mean"): - "Mean+whiskers" - displays the mean in a dark shade, one standard deviation around the mean in a medium shade, and the maximum/minimum in a light shade. For bar graphs only the mean, the mean plus a standard deviation, and the max are visible. This mode is not available if the Overlay method is stacked. - "Maximum" - displays the maximum value of all the points to be combined. - "Mean" - displays the mean. - "Minimum" - displays the minimum of all the points to be combined. Smoothing window: When set to a numerical value, this option determines the size, in pixels, of a smoothing window to be passed over the plot to smooth the edges of the bars or lines. This is equivalent to a trend line calculation on the graph. The default setting is "OFF". Negate values: When checked, all values in the wiggle are negated, meaning that positive values become negative and vice-versa. This is useful for wiggles representing transcription or other activities on the minus strand. Be aware that wiggles with negative values are drawn in altColor not color as positive values are. The below image shows ENCODE RNA-seq data around two genes on different strands, SIRT1 and HERC4, with a minus signal track using Negate values to flip the wiggle display to emphasize that HERC4 is expressed on the minus strand. This image also displays the signal graphed in points and a smoothing window of 16 pixels. [] Draw y indicator lines (boxed in orange): - at y= 0.0: Select the "ON" setting to display a line marking the 0.0 position on the graph (default is "OFF"). - at y= : Select the "ON" setting to display a line on the graph at the specified numerical value (defaults are "0" and "OFF"). This line can be used to mark a significant threshold on the graph. For example, in the image below, y=3. [] When you have finished making your configuration changes, click the Apply button to preview your changes, or click the Submit button to return to the annotation track display page. Annotation track display modes Each annotation track within the window may have up to five display modes: Dense The track is displayed with all features collapsed into a single line. The darker the line color the greater the wiggle value at that location. [Wiggle in dense display] Squish The track is displayed with all the features collapsed into a single line, much like the dense display mode with greater compression. [] Pack The track displays the wiggle value associated with each annotation feature creating a histogram-like image, much like the full display mode with greater compression. [] Full The track displays the wiggle value associated with each annotation feature creating a histogram-like image. [] Hide The track is not displayed at all. To hide all the annotation tracks, click the hide all button. Overlay method Note that not all graph-based tracks include the Overlay options. Transparent This setting displays the colored transparent graphs of multiple subtracks overlayed in the same vertical space. [] Solid This setting displays the colored opaque graphs of multiple subtracks overlayed in the same vertical space. [] Stacked This setting displays each graph stacked on top of each other where the high point of the graph is the sum of all the values. [] None This setting displays each graph in its own vertical space. [] 
/goldenPath/help/bigRmsk.html:Genome_Browser_bigRmsk_RepeatMasker_Format	bigRmsk Track Format The bigRmsk format allows for the display of annotations of a genome generated by the RepeatMasker program that screens DNA sequences for interspersed repeats and low complexity DNA sequences. It is the recommended method of adding RepeatMasker tracks to assembly hubs. The bigRmsk format enables taking the annotation output of RepeatMasker and converting it into a compressed and indexed bigBed file. Please see this page for a details of the bigBed format, its use, and associated tools. Display Conventions and Configuration Context Sensitive Zooming This track employs a technique which chooses the appropriate visual representation for the data based on the zoom scale and the number of annotations currently in view. The track will automatically switch from the most detailed visualization ('Full' mode) to the denser view ('Pack' mode) when the window size is greater than 45kb of sequence. It will further switch to the even denser single line view ('Dense' mode) if more than 500 annotations are present in the current view. Dense Mode Visualization In dense display mode, a single line is displayed denoting the coverage of repeats using a series of colored boxes. The boxes are colored based on the classification of the repeat (see below for legend). [] Pack Mode Visualization In pack mode, repeats are represented as sets of joined features. These are color coded as above based on the class of the repeat, and the further details such as orientation (denoted by chevrons) and a family label are provided. This family label may be optionally turned off in the track configuration. [] The pack display mode may also be configured to resemble the original UCSC repeat track. In this visualization, repeat features are grouped by classes (see below), and displayed on separate track lines. The repeat ranges are denoted as grayscale boxes, reflecting both the size of the repeat and the amount of base mismatch, base deletion, and base insertion associated with a repeat element. The higher the combined number of this divergence from the reference, the lighter the shading. [] Full Mode Visualization In the most detailed visualization, repeats are displayed as chevron boxes, indicating the size and orientation of the repeat. The interior grayscale shading represents the divergence of the repeat (see above) while the outline color represents the class of the repeat. Dotted lines above the repeat and extending left or right indicate the length of unaligned repeat model sequence and provide context for where a repeat fragment originates in its consensus or pHMM model. If the length of the unaligned sequence is large, an interruption line and bp size is indicated instead of drawing the extension to scale. [] For example, the following repeat is a SINE element in the forward orientation with average divergence. Only the 5' proximal fragment of the consensus sequence is aligned to the genome. The 3' unaligned length (384bp) is not drawn to scale and is instead displayed using a set of interruption lines along with the length of the unaligned sequence. [] Repeats that have been fragmented by insertions or large internal deletions are now represented by join lines. In the example below, a LINE element is found as two fragments. The solid connection lines indicate that there are no unaligned consensus bases between the two fragments. Also note these fragments form the 3' extremity of the repeat, as there is no unaligned consensus sequence following the last fragment. [] In cases where there is unaligned consensus sequence between the fragments, the repeat will look like the following. The dotted line indicates the length of the unaligned sequence between the two fragments. In this case the unaligned consensus is longer than the actual genomic distance between these two fragments. [] If there is consensus overlap between the two fragments, the joining lines will be drawn to indicate how much of the left fragment is repeated in the right fragment. [] The following table lists the repeat class colors: Color Repeat Class SINE - Short Interspersed Nuclear Element LINE - Long Interspersed Nuclear Element LTR - Long Terminal Repeat DNA - DNA Transposon Simple - Single Nucleotide Stretches and Tandem Repeats Low_complexity - Low Complexity DNA Satellite - Satellite Repeats RNA - RNA Repeats (including RNA, tRNA, rRNA, snRNA, scRNA, srpRNA) Other - Other Repeats (including class RC - Rolling Circle) Unknown - Unknown Classification A "?" at the end of the "Family" or "Class" (for example, DNA?) signifies that the curator was unsure of the classification. At some point in the future, either the "?" will be removed or the classification will be changed. bigRmsk track definitions The bigRmsk tracks consist of two bigBed files defined by autoSql schema: - The primary bigRmsk file, defined by bigRmskBed.as, which has the annotations of repeats. - The secondary bigRmskAlign file, defined by bigRmskAlignBed.as, which contains the alignments of the consensus repeats to the genome. This file is optional, if omitted, the bigRmsk track will function without the ability to view the alignments. The input files for the bigRmsk files are created from the RepeatMasker *.out and *.align files using the rmToTrackHub.pl program that is included with RepeatMasker. The bigRmsk format is not designed to work with any other type of data. Creating a bigRmsk track To create a bigRmsk track and its supporting files, follow the below steps. This assumes that you have already run RepeatMasker and have a *.out, and optionally *.align file. RepeatMasker output files are converted to the bigRmsk textual form using the RepeatMasker/util/rmToTrackHub.pl program that is part of the RepeatMasker 4.1.3 or newer distribution. Step 1. If you wish to experiment with quickly building an example track, download the example RepeatMasker output files for the human GRCh38 (hg38) assembly bigRmskExample.out and bigRmskExample.align used in this tutorial: wget https://genome.ucsc.edu/goldenPath/help/examples/bigRmskExample.out wget https://genome.ucsc.edu/goldenPath/help/examples/bigRmskExample.align Otherwise, substitute your *.out and *.align in theses instructions. Generating the alignment bigRmsk file is optional if you don't have the *.align files from RepeatMasker, the track will function with reduced functionality without them. Just skip the steps involved in build the alignment files. Step 2. Download the autoSql schemes bigRmskBed.as and bigRmskAlignBed.as: wget https://genome.ucsc.edu/goldenPath/help/examples/bigRmskBed.as wget https://genome.ucsc.edu/goldenPath/help/examples/bigRmskAlignBed.as You will also need a file of chromosome sizes for your genome, or download the hg38 file for the example: wget http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.chrom.sizes Step 3. Convert the RepeatMasker files to the text format bigRmsk files for conversion to the bigRmsk files with rmToTrackHub.pl, which sorts the output for direct input to bedToBigBed: RepeatMasker/util/rmToTrackHub.pl -out bigRmskExample.out -align bigRmskExample.align Step 4. Build the bigRmsk and optional bigRmskAlign files: bedToBigBed -tab -type=bed9+5 -as=bigRmskBed.as bigRmskExample.join.tsv hg38.chrom.sizes bigRmskExample.bb bedToBigBed -tab -type=bed3+14 -as=bigRmskAlignBed.as bigRmskExample.align.tsv hg38.chrom.sizes bigRmskExampleAlign.bb Step 6. Place the newly created bigRmsk file (bigRmskExample.bb), and optional bigRmskAlign (bigRmskExampleAlign.bb) to a web-accessible http, https or ftp location. Step 7. As with other bigBed-based tracks, bigRmsk tracks can be displayed as custom tracks, included in track hubs, or assembly hubs. The following options are used for bigRmsk custom tracks (with an equals sign between key and value) or trackDb hub entries as below: - type bigRmsk - bigDataUrl <url> - URL or relative path of bigRmsk file - xrefDataUrl <url> - URL or relative path of optional bigRmskAlign file A standard bigRmsk track description is available at bigRmskTrackDesc.html, which can be directly linked to with the URL: http://genome.ucsc.edu/goldenPath/trackDescriptions/bigRmskTrackDesc.html. See the Examples section below for detailed examples of bigRmsk custom tracks and track hub definitions. Examples Example of a bigRmsk custom track Construct a custom track using a single track line. Note that any of the track attributes listed here are applicable to tracks of type bigBed. To create a custom track using the example bigRmsk file: 1. Construct a track line that references the file: track type=bigRmsk name="bigRmsk Example" description="RepeatMasker example" visibility=full bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigRmskExample.bb xrefDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigRmskExampleAlign.bb 2. Paste the track line into the custom track management page for the human assembly hg38 (Dec. 2013). 3. Click the "submit" button. 4. Navigate to chr1:8,890-35,190 to see the track. Example of a bigRmsk track hub This example can also be loaded in a Track or Assembly Hub trackDb.txt with a stanza such as the following: track bigRmskExample shortLabel Example bigRmsk longLabel This is an example bigRmsk Track Hub Stanza type bigRmsk maxWindowToDraw 10000000 visibility full html http://genome.ucsc.edu/goldenPath/trackDescriptions/bigRmskTrackDesc.html bigDataUrl http://genome.ucsc.edu/goldenPath/help/examples/bigRmskExample.bb xrefDataUrl http://genome.ucsc.edu/goldenPath/help/examples/bigRmskExampleAlign.bb Additional information See the bigBed documentation for guidance on sharing, troubleshooting, and extracting data from bigRmsk files. Credits The bigRmsk system was developed by Robert Hubley of the Institute for Systems Biology. 
/goldenPath/help/vcf.html:Genome_Browser_VCF+tabix_Track_Format	VCF+tabix Track Format Variant Call Format (VCF) is a flexible and extendable line-oriented text format developed by the 1000 Genomes Project (now maintained by the GA4GH) for releases of single nucleotide variants, indels, copy number variants and structural variants discovered by the project. When a VCF file is compressed and indexed using tabix, and made web-accessible, the Genome Browser is able to fetch only the portions of the file necessary to display items in the viewed region. This makes it possible to display variants from files that are so large that the connection to UCSC would time out when attempting to upload the whole file to UCSC. Both the VCF file and its tabix index file remain on your web-accessible server (http, https, or ftp), not on the UCSC server. UCSC temporarily caches the accessed portions of the files to speed up interactive display. If you do not have access to a web-accessible server and need hosting space for your VCF files, please see the Hosting section of the Track Hub Help documentation. The UCSC tools support VCF versions 3.3 and greater. If you have VCF trio data, you may be interested in formatting your track as a Phased Trios track as described below. VCF Format VCF is an all-purpose format for defining variants of all types: SNVs, CNVs and translocations relative to a reference assembly. It can annotate all the variants in an individual as well as a population. Typically, a VCF file is too large to load directly into a custom track on the Browser and must be loaded as binary tabix-indexed file as described below. The full specification of VCF is found in documents on github. Here is a look at an example from that file showing a few rows of data for three samples. Details and descriptions of the data fields are in the .pdf. The data here can be pasted directly into hg18 human assembly on the Genome Browser. We have added two lines at the top of the entry that are not in the official example to make the display work in the Browser. Note that the first data field, identifying the chromosome, is in official VCF format which does not include the "chr" usually associated with Genome Browser chrom names. Either version will work in the Browser. track type=vcf name="vcf example" description="three samples in a vcf" db=hg18 visibility="full" browser position chr20:1-1306000 ##fileformat=VCFv4.2 ##fileDate=20090805 ##source=myImputationProgramV3.1 ##reference=file:///seq/references/1000GenomesPilot-NCBI36.fasta ##contig=<ID=20,length=62435964,assembly=B36,md5=f126cdf8a6e0c7f379d618ff66beb2da,species="Homo sapiens",taxonomy=x> ##phasing=partial ##INFO=<ID=NS,Number=1,Type=Integer,Description="Number of Samples With Data"> ##INFO=<ID=DP,Number=1,Type=Integer,Description="Total Depth"> ##INFO=<ID=AF,Number=A,Type=Float,Description="Allele Frequency"> ##INFO=<ID=AA,Number=1,Type=String,Description="Ancestral Allele"> ##INFO=<ID=DB,Number=0,Type=Flag,Description="dbSNP membership, build 129"> ##INFO=<ID=H2,Number=0,Type=Flag,Description="HapMap2 membership"> ##FILTER=<ID=q10,Description="Quality below 10"> ##FILTER=<ID=s50,Description="Less than 50% of samples have data"> ##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype"> ##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality"> ##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Read Depth"> ##FORMAT=<ID=HQ,Number=2,Type=Integer,Description="Haplotype Quality"> #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT NA00001 NA00002 NA00003 20 14370 rs6054257 G A 29 PASS NS=3;DP=14;AF=0.5;DB;H2 GT:GQ:DP:HQ 0|0:48:1:51,51 1|0:48:8:51,51 1/1:43:5:.,. 20 17330 . T A 3 q10 NS=3;DP=11;AF=0.017 GT:GQ:DP:HQ 0|0:49:3:58,50 0|1:3:5:65,3 0/0:41:3 20 1110696 rs6040355 A G,T 67 PASS NS=2;DP=10;AF=0.333,0.667;AA=T;DB GT:GQ:DP:HQ 1|2:21:6:23,27 2|1:2:0:18,2 2/2:35:4 20 1230237 . T . 47 PASS NS=3;DP=13;AA=T GT:GQ:DP:HQ 0|0:54:7:56,60 0|0:48:4:51,51 0/0:61:2 20 1234567 microsat1 GTC G,GTCT 50 PASS NS=3;DP=9;AA=G GT:GQ:DP 0/1:35:4 0/2:17:2 1/1:40:3 Generating a VCF track The typical workflow for generating a VCF custom track is this: 1. If you haven't done so already, download and build the tabix and bgzip programs. Test your installation by running tabix with no command-line arguments; it should print a brief usage message. For help with tabix, please contact the samtools-help mailing list (tabix is part of the samtools project). 2. Create VCF or convert another format to VCF. Items must be sorted by genomic position. 3. Compress your .vcf file using the bgzip program: bgzip my.vcf For more information about the bgzip command, run it with no arguments to display the usage message. 4. Create a tabix index file (.tbi or .csi) for the bgzip-compressed VCF (.vcf.gz). By default, the tabix command appends .tbi to the my.vcf.gz filename, creating a binary index file named my.vcf.gz.tbi with which genomic coordinates can quickly be translated into file offsets in my.vcf.gz: tabix -p vcf my.vcf.gz The tabix (.tbi) and BAI index formats can handle individual chromosomes up to 512 Mbp (2^29 bases) in length. If your input file contains data lines with start or end positions greater than 512 Mbp, you will need to use a CSI (.csi) index instead. tabix --csi -p vcf my.vcf.gz 5. Move both the compressed VCF file (my.vcf.gz) and index file (my.vcf.gz.tbi or my.vcf.gz.csi) to an http, https, or ftp location. Note that the Genome Browser looks for an index file with the same URL as the VCF file with the .tbi or .csi suffix added. If your hosting site does not use the filename as the URL link, you will have to specifically call the location of this .vcf.tbi/csi index file with the bigDataIndex keyword. You can read more about bigDataIndex in the TrackDb Database Definition page. 6. Construct a custom track using a single track line. The basic version of the track line will look something like this: track type=vcfTabix name="My VCF" bigDataUrl=http://myorg.edu/mylab/my.vcf.gz Again, in addition to http://myorg.edu/mylab/my.vcf.gz, the associated index file http://myorg.edu/mylab/my.vcf.gz.tbi must also be available at the same location. If the file is in a different location or uses a different filename, then use the bigDataIndex attribute in the track line to point to the index file. track type=vcfTabix name="My VCF" bigDataUrl=http://myorg.edu/mylab/my.vcf.gz bigDataIndex=http://myorg.edu/someOtherDirectory/myvcf.gz.tbi 7. Paste the custom track line into the text box in the custom track management page, click "submit" and view in the Genome Browser. Parameters for VCF custom track definition lines All options are placed in a single line separated by spaces (lines are broken only for readability here): track type=vcfTabix bigDataUrl=http://... hapClusterEnabled=true|false hapClusterColorBy=altOnly|refAlt|base hapClusterTreeAngle=triangle|rectangle hapClusterHeight=N applyMinQual=true|false minQual=Q minFreq=F name=track_label description=center_label visibility=display_mode priority=priority db=db maxWindowToDraw=N chromosomes=chr1,chr2,... Note if you copy/paste the above example, you must remove the line breaks. Click here for a text version that you can paste without editing. The track type and bigDataUrl are REQUIRED: type=vcfTabix bigDataUrl=http://myorg.edu/mylab/my.vcf.gz The remaining settings are OPTIONAL. Some are specific to VCF: hapClusterEnabled true|false # if file has phased genotypes, sort by local similarity hapClusterColorBy altOnly|refAlt|base # coloring scheme, default altOnly, conditional on hapClusterEnabled hapClusterTreeAngle triangle|rectangle # draw leaves as < or [, default <, conditional on hapClusterEnabled hapClusterHeight N # height of track in pixels, default 128, conditional on hapClusterEnabled applyMinQual true|false # if true, don't display items with QUAL < minQual; default false minQual Q # minimum value of Q column to display item, conditional on applyMinQual minFreq F # minimum minor allele frequency to display item; default 0.0 Other optional settings are not specific to VCF, but relevant: name track label # default is "User Track" description center label # default is "User Supplied Track" visibility squish|pack|full|dense|hide # default is hide (will also take numeric values 4|3|2|1|0) priority N # default is 100 db genome database # e.g. hg19 for Human Feb. 2009 (GRCh37) maxWindowToDraw N # don't display track when viewing more than N bases chromosomes chr1,chr2,... # track contains data only on listed reference assembly sequences The VCF track configuration help page describes the VCF track configuration page options. Phased Trio format The vcfPhasedTrio track type is available for users whose VCF contains genotype data from one to three individuals. The underlying VCF follows the standard VCF format as described above, with the added caveat that there must be GENOTYPE columns for each of the individuals present. An example of the trio display is show below for the 1000 Genomes Trio track on Human/GRCh38: [3 VCF Phased Trio tracks along with the GENCODE v32 genes from the Human/GRCh38 assembly. Each of two diploid haplotypes for each individual in a trio is drawn as a black lane, with snps as vertical ticks on the haplotype they fall on. Ticks are shaded blue,red,green or black according to their predicted functional effect.] Unlike a regular genome browser track, Trio tracks display the genome variants of each individual as two haplotypes; SNPs, small insertions and deletions are mapped to each haplotype based on the phasing information of the VCF file. Each haplotype is displayed on two separate, horizontal black lines across the browser window. Each variant is drawn as a vertical dash. Homozygous variants will show two identical dashes on both haplotype lines. Phased heterozygous variants are placed on one of the haplotype lanes and unphased heterozygous variants are displayed in the area between the two haplotype lines. Follow the steps for a normal VCF file, including moving the file to a web accessible location and generating a tabix index file, then use the following required vcfPhasedTrio trackDb settings to view the trio display: type vcfPhasedTrio # The track type is required and must be "vcfPhasedTrio" bigDataUrl http://url.to.vcfFile # The bigDataUrl is required vcfChildSample GT ID|alias # the Genotype column ID of the "child" sample, with an optional "|" followed by a human readable alias for the ID There are also two optional settings for vcfPhasedTrio tracks: vcfParentSamples GT ID1|alias1,GT ID2|alias2 # comma separated (no spaces) list of the "parent" samples, with optional aliases vcfUseAltSampleNames GT ID # Use the aliases in the display by default instead of the Genotype column ID Other optional settings are not specific to VCF, but relevant: maxWindowToDraw N # don't display track when viewing more than N bases chromosomes chr1,chr2,... # track contains data only on listed reference assembly sequences Examples Example #1 In this example, you will create a custom track for an indexed VCF file that is already on a public server — variant calls generated by the 1000 Genomes Project. The line breaks inserted here for readability must be removed before submitting the track line: browser position chr21:33,034,804-33,037,719 track type=vcfTabix name="VCF Example One" description="VCF Ex. 1: 1000 Genomes phase 1 interim SNVs" chromosomes=chr21 maxWindowToDraw=200000 db=hg19 visibility=pack bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/vcfExample.vcf.gz The "browser" line above is used to view a small region of chromosome 21 with variants from the .vcf.gz file. Note if you copy/paste the above example, you must remove the line breaks (or, click here for a text version that you can paste without editing). Paste the "browser" line and "track" line into the custom track management page for the human assembly hg19 (Feb. 2009), then click the "submit" button. On the following page, click the "chr21" link in the custom track listing to view the VCF track in the Genome Browser. Example #2 In this example, you will create compressed, indexed VCF from an existing VCF text file. First, save this VCF file -- vcfExampleTwo.vcf -- to your machine. Perform steps 1 and 3-7 in the workflow described above, but substitute vcfExampleTwo.vcf for my.vcf. On the custom track management page, click the "add custom tracks" button if necessary and make sure that the genome is set to "Human" and the assembly is set to "Feb. 2009 (hg19) " before pasting the track line and submitting. Remember to remove the line breaks that have been added to the track line for readability (or, click here for a text version that you can paste without editing): track type=vcfTabix name="VCF Example Two" bigDataUrl=http://myorg.edu/mylab/vcfExampleTwo.vcf.gz description="VCF Ex. 2: More variants from 1000 Genomes" visibility=pack db=hg19 chromosomes=chr21 browser position chr21:33,034,804-33,037,719 browser pack snp132Common Example #3 In this example, you will load a hub that has VCF data described in a hub's trackDb.txt file. First, navigate to the Basic Hub Quick Start Guide and review an introduction to hubs. Visualizing VCF files in hubs involves creating three text files: hub.txt, genomes.txt, and trackDb.txt. The browser is passed a URL to the top-level hub.txt file that points to the related genomes.txt and trackDb.txt files. The trackDb.txt file contains stanzas for each track that outlines the details and type of each track to display, such as these lines for a VCF file located at the bigDataUrl location: track vcf1 bigDataUrl http://hgdownload.soe.ucsc.edu/gbdb/hg19/1000Genomes/ALL.chr21.integrated_phase1_v3.20101123.snps_indels_svs.genotypes.vcf.gz #Note: there is a corresponding fileName.vcf.gz.tbi in the same above directory shortLabel chr21 VCF example longLabel This chr21 VCF file is an example from the 1000 Genomes Phase 1 Integrated Variant Calls Track type vcfTabix visibility dense Note: there is now a useOneFile on hub setting that allows the hub properties to be specified in a single file. More information about this setting can be found on the Genome Browser User Guide. Here is a direct link to the trackDb.txt file to see more information about this example hub, and below is a direct link to visualize the hub in the browser, where this example VCF file displays in dense mode alongside the other tracks in this hub. You can find more Track Hub VCF display options on the Track Database (trackDb) Definition Document page. http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg19&hubUrl=http://genome.ucsc.edu/goldenPath/help/examples/hubDirectory/hub.txt Sharing Your Data with Others If you would like to share your VCF data track with a colleague, learn how to create a URL by looking at Example 6 on the custom tracks page. 
/goldenPath/help/bigGenePred.html:Genome_Browser_bigGenePred_Track_Format Genome_Browser_bigGenePred_Track_Format	bigGenePred Track Format The bigGenePred format stores positional annotations for collections of exons in a compressed format, similar to how BED files are compressed into bigBeds. The bigGenePred format is a superset of the genePred text-based format supported using the bigBed format, so it can be efficiently accessed over a network. The bigGenePred format includes 8 additional fields that contain details about coding frames, annotation status, and other gene-specific information. This is commonly used in the Browser to display start codons, stop codons, and amino acid translations. Before compression, bigGenePred files can be described as bed12+8 files. bigGenePred files can be created using the program bedToBigBed, run with the -as option to pull in a special autoSql (.as) file that defines the extra fields of the bigGenePred. Much like bigBed, bigGenePred files are in an indexed binary format. The advantage of using a binary format is that only the portions of the file needed to display a particular region are read, allowing for much faster performance when working with large data sets. As with all big* files, bigGenePred files must be hosted on a web-accessible server (http, https, or ftp) to be displayed. For more information on finding a hosting location for your bigGenePred files, please see the hosting section of the Track Hub Help documentation. bigGenePred format description The following autoSql definition specifies bigGenePred gene prediction files. This definition, contained in the file bigGenePred.as, is pulled in when the bedToBigBed utility is run with the -as=bigGenePred.as option. table bigGenePred "bigGenePred gene models" ( string chrom; "Reference sequence chromosome or scaffold" uint chromStart; "Start position in chromosome" uint chromEnd; "End position in chromosome" string name; "Name or ID of item, ideally both human-readable and unique" uint score; "Score (0-1000)" char[1] strand; "+ or - for strand" uint thickStart; "Start of where display should be thick (start codon)" uint thickEnd; "End of where display should be thick (stop codon)" uint reserved; "RGB value (use R,G,B string in input file)" int blockCount; "Number of blocks" int[blockCount] blockSizes; "Comma separated list of block sizes" int[blockCount] chromStarts;"Start positions relative to chromStart" string name2; "Alternative/human readable name" string cdsStartStat; "Status of CDS start annotation (none, unknown, incomplete, or complete)" string cdsEndStat; "Status of CDS end annotation (none, unknown, incomplete, or complete)" int[blockCount] exonFrames; "Reading frame of the start of the CDS region of the exon, in the direction of transcription (0,1,2), or -1 if there is no CDS region." string type; "Transcript type" string geneName; "Primary identifier for gene" string geneName2; "Alternative/human-readable gene name" string geneType; "Gene type" ) The field exonFrames is a comma-separated list of the numbers with the possible values 0, 1, 2 or -1, one per exon, in order of transcription. This order means that the first value for a transcript on the minus (-) strand is the exon on the right of the screen on the Genome Browser. A value of zero means that the first codon of the exon starts at the first nucleotide of the exon. A value of one means that the first codon starts after the first nucleotide and a value of two means that it starts after the second nucleotide. UTRs are non-coding and their exonFrame value is -1. The fields cdsStartStat and cdsEndStat have the following values: 'none' = none, 'unk' = unknown, 'incmpl' = incomplete, and 'cmpl' = complete. The values, however, are not used for our display and cannot be used to identify coding or non-coding genes. For most purposes, to get more information about a transcript, other tables will need to be used. For instance, in the case of hg38, the tables named wgEncodeGencodeAttrsVxx, where xx is the Gencode Version number. See this coding/non-coding genes FAQ for more information. The following bed12+8 is an example of a pre-bigGenePred text file . Creating a bigGenePred track from a bed12+8 file Step 1. Format your pre-bigGenePred file. The first 12 fields of pre-bigGenePred files are described by the BED file format. Your file must also contain the 8 extra fields described in the autoSql file definition shown above: name2, cdsStartStat, cdsEndStat, exonFrames, type, geneName, geneName2, geneType. For example, you can use this bed12+8 input file, bigGenePred.txt. Your pre-bigGenePred file must be sorted first on the chrom field, and secondarily on the chromStart field. You can use the UNIX sort command to do this: sort -k1,1 -k2,2n unsorted.bed > input.bed Step 2. Download the bedToBigBed program from the binary utilities directory. Step 3. Download the chrom.sizes file for your assembly from our downloads page (click on "Full data set" for your organism). For example, the hg38.chrom.sizes file for the hg38 database is located at http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.chrom.sizes. Alternatively, you can use the fetchChromSizes script from the utilities directory. Step 4. Create the bigGenePred file from your pre-bigGenePred file using the bedToBigBed utility command: bedToBigBed -as=bigGenePred.as -type=bed12+8 bigGenePred.txt chrom.sizes myBigGenePred.bb Step 5. Move the newly created bigGenePred file (myBigGenePred.bb) to a web-accessible http, https, or ftp location. See hosting section if necessary. Step 6. Construct a custom track using a single track line. Any of the track attributes will be available for use on bigBed tracks. The basic version of the track line will look something like this: track type=bigGenePred name="My Big GenePred" description="A Gene Set Built from Data from My Lab" bigDataUrl=http://myorg.edu/mylab/myBigGenePred.bb Step 7. Paste this custom track line into the text box on the custom track page with your modified URL. Click Submit and your track should load successfully. Then click Go to be taken to the Browser window with your custom track at the top. Note that there might not be data at all positions. Examples Example #1: Create a Custom Track Create a bigGenePred custom track using the bigGenePred file located on the UCSC Genome Browser http server, bigGenePred.bb. This file contains data for the hg38 assembly. 1. Construct a track line that references the hosted file: track type=bigGenePred name="bigGenePred Example One" description="A bigGenePred file" bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigGenePred.bb 2. Paste the track line into the custom track page for the human assembly, hg38. 3. Click the Submit button. Custom tracks can also be loaded via one URL line. The link below loads the same bigGenePred track and sets additional parameters in the URL: http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&hgct_customText=track%20type=bigGenePred%20bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigGenePred.bb After this example bigGenePred track is loaded in the Genome Browser, click on the track to change display from dense to pack, then click on a gene in the Browser's track display to view the gene details page. Note that the page offers links to translated protein, predicted mRNA, and genomic sequence. Example #2: Display Amino Acids and Codon Numbers In this example, you will configure the bigGenePred track loaded in Example #1 to display amino acids and codon numbering: 1. Access the track configuration page by right-clicking anywhere in the track and clicking "Configure User Track" or alternately, from within a gene's details page, click the "Go to User Track track controls" link. 2. Making sure the display is in pack or full visibility mode, change the "Color track by codons:" option from "OFF" to "genomic codons". Then click Submit or Ok . 3. Zoom to a region with track data, such as chr9:133,255,650-133,255,700, and note that the track now displays amino acids. 4. Return to the track configuration page and check the box next to "Show codon numbering", then click Submit . The Browser tracks display will now show amino acid letters and codon numbering when sufficiently zoomed in. Alternatively, you can also add a parameter in the custom track line, baseColorDefault=genomicCodons, to set amino acids and codon numbering to display by default: browser position chr10:67,884,600-67,884,900 track type=bigGenePred baseColorDefault=genomicCodons name="bigGenePred Example Two" description="A bigGenePred file" visibility=pack bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigGenePred.bb Paste the above into the hg38 custom track page to view an example of bigGenePred amino acid display at the beginning of the SIRT1 gene on chromosome 10. [An image of a track with codons colored] Example #3: Bed12+8 to BigGenePred In this example, you will create your own bigGenePred file from an existing pre-bigGenePred input file, a bed12+8 file. 1. Save the example bed12+8 input file to your computer, bigGenePred.txt. 2. Download the bedToBigBed utility (Step 2, in the Creating a bigGenePred section above). 3. Save the hg38.chrom.sizes text file to your computer. This file contains the chrom.sizes for the human hg38 assembly (Step 3, above). 4. Save the autoSql file bigGenePred.as to your computer. 5. Run the bedToBigBed utility to create the bigGenePred output file (step 4, above): bedToBigBed -type=bed12+8 -tab -as=bigGenePred.as bigGenePred.txt hg38.chrom.sizes bigGenePred.bb 6. Place the newly created bigGenePred file (bigGenePred.bb) on a web-accessible server (Step 5, above). 7. Construct a track line that points to the bigGenePred file (Step 6, above). 8. Create the custom track on the human assembly hg38 (Dec. 2013), and view it in the Genome Browser (step 7, above). Example #4: GTF (or GFF) to BigGenePred In this example, you will convert a GTF file to bigGenePred using command line utilities. You will need gtfToGenePred, genePredTobigGenePred, and bedToBigBed. If you would like to convert a GFF file to bigGenePred, you will use gff3ToGenePred in place of the gtfToGenePred. You can download utilities from the utilities directory. 1. Obtain a GTF file using the wget command. Skip this step if you already have a GTF or GFF file. wget http://genome.ucsc.edu/goldenPath/help/examples/bigGenePredExample4.gtf 2. Convert the GTF file to genePred extended format using the gtfToGenePred command. gtfToGenePred -genePredExt bigGenePredExample4.gtf example4.genePred If you are converting a GFF file, use the gff3ToGenePred command. gff3ToGenePred yourFile.gff example4.genePred 3. Convert the genePred extended file to a pre-bigGenePred text file. genePredToBigGenePred example4.genePred bigGenePredEx4.txt 4. Download a helper file that specifies column names. wget https://genome.ucsc.edu/goldenPath/help/examples/bigGenePred.as 5. Convert your text bigGenePred to a binary indexed format. If you are not using hg38, you will need to replace the hg38.chrom.sizes file path with your organism's file path from the downloads directory under "Genome Sequence Files". bedToBigBed -type=bed12+8 -tab -as=bigGenePred.as bigGenePredEx4.txt http://hgdownload.soe.ucsc.edu/goldenPath/hg38/bigZips/hg38.chrom.sizes bigGenePredEx4.bb 6. Put your binary indexed file, bigGenePredEx4.bb, in a web-accessible location. See the hosting section for more information. 7. To view this example, you can click this into this Browser link. To view your own data, paste the link into your web browser and replace the URL after "bigDataUrl=" with a link to your own hosted data. http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&position=chr19:44905790-44909388&hgct_customText=track%20type=bigGenePred%20bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigGenePredEx4.bb You can also add your data in the custom track management page. This allows you to set position, configuration options, and write a more complete desciption. If you want to see codons, you can right click, then click configure codon view or set this options using baseColorDefault=genomicCodons as is done below. browser position chr19:44905790-44909388 track type=bigGenePred baseColorDefault=genomicCodons name="bigGenePred Example Four" description="Ex4:BigGenePred Made from GTF" visibility=pack bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigGenePredEx4.bb Once you are done, you should have a track on the Genome Browser like the one below. [An image of a bigGenePred track on the Browser] Example #5: Create a BigGenePred Track Hub In this example, you will set up a Track Hub that displays bigGenePred data and uses one of the bigGenePred-specific settings to display gene codons. You can see a pre-built version of this hub by clicking this link. 1. Make sure you have access to a web-hosted file location like GitHub, CyVerse, or an institutional website. This is where you will store your bigData files and configuration files. For more information, please visit the hosting section of our Track Hub help guide. 2. Copy the text from the bigGenePred example hub files into identically named files on your hosted website. You will need a hub.txt and a genomes.txt file, along with a directory for each assembly you would like to visualize (hg38). This assembly directory stores your data files (bigGenePred.bb) and a trackDb.txt file which defines track settings, such as track name, description, and the bigGenePred setting for amino acid display (baseColorDefault genomicCodons). Visit the trackDb help page for more information about trackDb settings. 3. Verify that you have the four minimum files, two of which in the hg38 directory: --hub.txt --genomes.txt --hg38 ----trackDb.txt ----bigGenePred.bb You may also include a hub description page and a track description page, where you will find additional instructions. 4. Copy the URL address of your hub.txt file and paste it into the text input box on the Track Hub page. For an example, you can paste the following link into the input box: http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubBigGenePred/hub.txt. Clicking the Add Hub button will connect to your track hub and navagate you to your assembly of interest. For more instructions on setting up Track Hubs, visit the Track Hub set-up page. Sharing your data with others If you would like to share your bigGenePred data track with a colleague, learn how to create a URL link to your data by looking at Example #6 on the custom track help page. Extracting data from bigBed format Because the bigGenePred files are an extension of bigBed files, which are indexed binary files, it can be difficult to extract data from them. UCSC has developed the following programs to assist in working with bigBed formats, available from the binary utilities directory. - bigBedToBed — converts a bigBed file to ASCII BED format. - bigBedSummary — extracts summary information from a bigBed file. - bigBedInfo — prints out information about a bigBed file. As with all UCSC Genome Browser programs, simply type the program name (with no parameters) at the command line to view the usage statement. Troubleshooting If you encounter an error when you run the bedToBigBed program, check your input file for data coordinates that extend past the end of the chromosome. If these are present, run the bedClip program (available here) to remove the problematic row(s) before running the bedToBigBed program. 
/goldenPath/help/api.html:REST_API_data_interface	REST API data interface Contents Why you may not want to use this API What is REST? What is JSON output data? What is the access URL? What type of data can be accessed? Endpoint functions Parameters to endpoint functions Required and optional parameters Supported track types Using the API on mirrors and local installations Example data access, list functions Example data access, getData functions Example data access, Search functions Error return examples Practical examples Why you may not want to use this API Genomic data is considerably large, and computational biologists generally need all the data that is available for their analyses. Web APIs, as a technology, were designed for retrieving relatively small pieces of data, often from Javascript. Consequently, Web APIs may not be the best way to get data from the UCSC Genome Browser. For this reason, we also provide alternative methods to access our data: - The Table Browser for custom exports and reports as tab-separated files and can be joined with other tables. - For tracks that are stored in MySQL, our MySQL/MariaDB server is publicly accessible. - Dumps of SQL-stored tracks, as well as tracks in binary file formats, can be found on our download server. - All binary files on the download server (bigBed, bigWig and derivatives like bigPsl, bigChain, etc) can even be streamed over HTTP. All these options offer faster bulk downloads and are often easier to parse from scripting languages. If, however, you write Javascript clients or need only a few features within a given range, then the endpoints documented on this page may address your needs. If you require an option or endpoint not listed below for your specific use case, contact us and we can try and implement the feature. Conditions of use - Avoid excessive or heavy queries that may impact the server performance. Inappropriate query use will result in a restriction of access. - We generally recommend a maximum of one hit every second, but there is no strict limit on queries using the API. A botDelay system is in place to avoid overloads onto the system. - If you begin to notice a significant delay between requests, adjust your code or try an alternative method to retrieve our data. For more details about the Conditions of Use, please refer to the following page, Genome Browser Conditions of Use. If you plan to execute a query that you think may be excessive, contact UCSC first to avoid the possibility of temporarily restricting your access to the REST API. What are REST and JSON? REST is an acronym for REpresentational State Transfer. It states architectural guidelines for how an API will operate. See also: Principles of REST. Like most APIs, ours returns data in JSON format. JSON data is a data transfer syntax almost identical to the Javascript object and array syntax. See also: JSON Introduction What is the access URL? This access URL: https://api.genome.ucsc.edu/ is used to access the endpoint functions. For example: wget -O- 'https://api.genome.ucsc.edu/list/publicHubs' What type of data can be accessed? The following data sets can be accessed at this time: - List of available public hubs - List of available UCSC Genome Browser genome assemblies - List of files available for download for UCSC Browser genome assemblies - List of genomes from a specified assembly or track hub - List of available data tracks from a specified hub or UCSC Genome Browser genome assembly (see also: track definition help) - List of chromosomes contained in an assembly hub or UCSC Genome Browser genome assembly - List of chromosomes contained in a specific track of an assembly or track hub, or UCSC Genome Browser genome assembly - Return DNA sequence from an assembly hub 2bit file, or UCSC Genome Browser assembly - Return track data from a specified assembly or track hub, or UCSC Genome Browser assembly - Return search matches to words in track data, track names, track descriptions, public hub track names, and public hub descriptions within a UCSC Genome Browser genome assembly Note: BLAT also supports programmatic URL queries which return in JSON format. See our BLAT FAQ for more info. Endpoint functions to return data The URL https://api.genome.ucsc.edu/ is used to access the endpoint functions. For example: curl -L 'https://api.genome.ucsc.edu/list/ucscGenomes' - /list/publicHubs - list public hubs - /list/ucscGenomes - list UCSC Genome Browser database genomes from database host - /list/hubGenomes - list genomes from specified hub - /list/files - list download files available for specified genome - /list/tracks - list data tracks available in specified hub or database genome (see also: track definition help) - /list/chromosomes - list chromosomes from a data track in specified hub or database - /list/schema - list the schema for a data track in specified hub or database genome - /getData/sequence - return sequence from specified hub or database genome - /getData/track - return data from specified track in hub or database genome - /search - return search matches within a UCSC Genome Browser genome assembly Parameters to endpoint functions - hubUrl=<url> - specify track hub or assembly hub URL - genome=<name> - specify genome assembly in UCSC Genome Browser or track/assembly hub. Use with with /list/genarkGenomes to test for existence. - track=<trackName> - specify data track in track/assembly hub or UCSC database genome assembly - chrom=<chrN> - specify chromosome name for sequence or track data - start=<123> - specify start coordinate (0 relative) for data from track or sequence retrieval (start and end required together). See also: UCSC browser coordinate counting systems - end=<456> - specify end coordinate (1 relative) for data from track or sequence retrieval (start and end required together). See also: UCSC browser coordinate counting systems - revComp=1 - on /getData/sequence function, return reverse complement of sequence data - maxItemsOutput=1000 - limit number of items to output, default: 1,000, maximum limit: 1,000,000 (use -1 to get maximum output) - trackLeavesOnly=1 - on /list/tracks function, only show tracks, do not show composite container information - jsonOutputArrays=1 - on /getData/track function, JSON format is array type for each item of data, instead of the default object type - format=text - on /list/files function, return plain text listing of download files instead of JSON format output (which includes more meta-data information). Text output contains less meta-data in comment lines prefixed by the '#' hash character. - search=<term>&genome=<name> - on /search function, specify term to be search within a UCSC Genome Browser genome assembly - categories=helpDocs - on /search?search=<term>&genome=<name> function, restrict the search within the UCSC Genome Browser help documentation - categories=publicHubs - on /search?search=<term>&genome=<name> function, restrict the search within the UCSC Genome Browser Public Hubs - categories=trackDb - on /search?search=<term>&genome=<name> function, restrict the search within the track database (trackDb) settings The parameters are added to the endpoint URL beginning with a question mark ?, and multiple parameters are separated with the semi-colon ;. For example: https://api.genome.ucsc.edu/getData/sequence?genome=hg38;chrom=chrM Required and optional parameters Endpoint function Required Optional --------------------- ------------------------------------------- ---------------------------------------------------------------- /list/publicHubs (none) (none) /list/ucscGenomes (none) (none) /list/genarkGenomes (none) genome, maxItemsOutput /list/hubGenomes hubUrl (none) /list/files genome format=text, maxItemsOutput /list/tracks genome or (hubUrl and genome) trackLeavesOnly=1 /list/chromosomes genome or (hubUrl and genome) track /list/schema (genome or (hubUrl and genome)) and track (none) /getData/sequence (genome or (hubUrl and genome)) and chrom start, end, revComp=1 /getData/track (genome or (hubUrl and genome)) and track chrom, (start and end), maxItemsOutput, jsonOutputArrays /search search and genome categories=helpDocs, categories=publicHubs, categories=trackDb The hubUrl and genome parameters are required together to specify a unique genome in an assembly or track hub. The genome for a track hub will usually be a UCSC database genome. Assembly hubs will have their own unique genome sequences. Specify genome without a hubUrl to refer to a UCSC Genome Browser assembly. Using the chrom=<name> parameter will limit the request to the single specified chromosome. To limit the request to a specific position, both start=4321 and end=5678 must be given together. Using the revComp=1 parameter returns the reverse complement. Use the genome argument with the /list/genarkGenomes function to test for the existence of a specific genome assembly in the Genark set of assembly hubs. The /list/files endpoint only works for UCSC hosted genome assemblies, not for external hosted assembly hubs. Any extra parameters not allowed in a function will be flagged as an error. Supported track types for getData functions - altGraphX (e.g. 'sibTxGraph' for hg19) - barChart/bigBarChart - bed - bigBed - bigChain - bigGenePred - bigLolly - bigNarrowPeak - bigMaf - bigPsl - bigWig - chain - ctgPos (e.g. 'ctgPos2' for hg19) - expRatio (e.g. 'gnfAtlas2' for mm9) - factorSource (e.g. 'encRegTfbsClustered' for hg38) - genePred - gvf (e.g. 'cnvDevDelayCase' for hg19) - interact/bigInteract - narrowPeak - netAlign - peptideMapping - pgSnp - psl - rmsk - repeat masker - wiggle/wig Work is underway to support additional track types Using the API on mirrors and local installations In order to access the API from a mirror installation or one of the UCSC official mirrors, the complete URL with the cgi-bin should be used: - Europe: https://genome-euro.ucsc.edu/cgi-bin/hubApi/ - Asia: https://genome-asia.ucsc.edu/cgi-bin/hubApi/ - - Mirror installation: https://your.server.edu/cgi-bin/hubApi/ The URL can then be passed any of the functions described in this page: - list public hubs - https://genome-euro.ucsc.edu/cgi-bin/hubApi/list/publicHubs - list UCSC database genomes - https://genome-asia.ucsc.edu/cgi-bin/hubApi/list/ucscGenomes Example data access Your WEB browser can be configured to interpret JSON data and format in a convenient browsing format. Firefox has this function built in, other browsers have add-ons that can be turned on to format JSON data. With your browser thus configured, the following links can demonstrate the functions of the API interface. Listing functions 1. list public hubs - api.genome.ucsc.edu/list/publicHubs 2. list UCSC database genomes - api.genome.ucsc.edu/list/ucscGenomes 3. list GenArk assembly hub genomes - api.genome.ucsc.edu/list/genarkGenomes?maxItemsOutput=5 4. test if genome GCF_028858775.2 exists in the GenArk assembly hub genomes - api.genome.ucsc.edu/list/genarkGenomes?genome=GCF_028858775.2 5. list genomes from specified hub - api.genome.ucsc.edu/list/hubGenomes?hubUrl=http://hgdownload.soe.ucsc.edu/hubs/mouseStrains/hub.txt 6. list tracks from specified hub and genome - api.genome.ucsc.edu/list/tracks?hubUrl=http://hgdownload.soe.ucsc.edu/hubs/mouseStrains/hub.txt;genome=CAST_EiJ 7. list tracks from UCSC database genome - api.genome.ucsc.edu/list/tracks?genome=hg38 8. list chromosomes from UCSC database genome - api.genome.ucsc.edu/list/chromosomes?genome=hg38 9. list chromosomes from specified track in UCSC database genome - api.genome.ucsc.edu/list/chromosomes?genome=hg38;track=gold 10. list chromosomes from assembly hub genome - api.genome.ucsc.edu/list/chromosomes?hubUrl=http://hgdownload.soe.ucsc.edu/hubs/mouseStrains/hub.txt;genome=CAST_EiJ 11. list chromosomes from specified track in assembly hub genome - api.genome.ucsc.edu/list/chromosomes?hubUrl=http://hgdownload.soe.ucsc.edu/hubs/mouseStrains/hub.txt;genome=CAST_EiJ;track=assembly 12. list schema from specified track in UCSC database genome - api.genome.ucsc.edu/list/schema?genome=hg38;track=knownGene 13. list download files for UCSC GenArk genome 'GCF_000955945.1' in JSON format limit 5 items output - api.genome.ucsc.edu/list/files?genome=GCF_000955945.1;maxItemsOutput=5 14. list download files for UCSC genome 'hs1' in plain text format limit 5 items output - api.genome.ucsc.edu/list/files?genome=hs1;format=text;maxItemsOutput=5 getData functions 1. Get DNA sequence from specified chromosome in UCSC database genome - api.genome.ucsc.edu/getData/sequence?genome=hg38;chrom=chrM 2. Get DNA sequence from specified chromosome and start,end coordinates in UCSC database genome - api.genome.ucsc.edu/getData/sequence?genome=hg38;chrom=chrM;start=4321;end=5678 3. Get the reverse complement of the DNA sequence from specified chromosome and start,end coordinates in UCSC database genome - api.genome.ucsc.edu/getData/sequence?genome=hg38;chrom=chrM;start=4321;end=5678;revComp=1 4. Get DNA sequence from a track hub where 'genome' is a UCSC database - api.genome.ucsc.edu/getData/sequence?hubUrl=http://hgdownload.soe.ucsc.edu/hubs/mouseStrains/hub.txt;genome=mm10;chrom=chrM;start=4321;end=5678 5. Get DNA sequence from specified chromosome and start,end coordinates in an assembly hub genome - api.genome.ucsc.edu/getData/sequence?hubUrl=http://hgdownload.soe.ucsc.edu/hubs/mouseStrains/hub.txt;genome=CAST_EiJ;chrom=chr1;start=4321;end=5678 6. Get track data for specified track in UCSC database genome - api.genome.ucsc.edu/getData/track?genome=hg38;track=gold;maxItemsOutput=100 7. Get track data for specified track and chromosome in UCSC database genome - api.genome.ucsc.edu/getData/track?genome=hg38;track=gold;chrom=chrM 8. Get track data for specified track, chromosome and start,end coordinates in UCSC database genome - api.genome.ucsc.edu/getData/track?genome=hg38;track=gold;chrom=chr1;start=47000;end=48000 9. Get track data for specified track in an assembly hub genome - api.genome.ucsc.edu/getData/track?hubUrl=http://hgdownload.soe.ucsc.edu/hubs/mouseStrains/hub.txt;genome=CAST_EiJ;track=assembly 10. Get track data for specified track and chromosome in an assembly hub genome - api.genome.ucsc.edu/getData/track?hubUrl=http://hgdownload.soe.ucsc.edu/hubs/mouseStrains/hub.txt;genome=CAST_EiJ;track=assembly;chrom=chr1 11. Get track data for specified track in a track hub - api.genome.ucsc.edu/getData/track?hubUrl=http://hgdownload.soe.ucsc.edu/hubs/mouseStrains/hub.txt;genome=CAST_EiJ;track=ensGene 12. Get track data for specified track and chromosome in a track hub - api.genome.ucsc.edu/getData/track?hubUrl=http://hgdownload.soe.ucsc.edu/hubs/mouseStrains/hub.txt;genome=CAST_EiJ;track=ensGene;chrom=chr1 13. Wiggle track data for specified track, chromosome with start and end limits in an assembly hub genome - api.genome.ucsc.edu/getData/track?hubUrl=http://hgdownload.soe.ucsc.edu/hubs/mouseStrains/hub.txt;genome=CAST_EiJ;track=gc5Base;chrom=chr1;start=4321;end=5678 14. Wiggle track data for specified track in a UCSC database genome - api.genome.ucsc.edu/getData/track?genome=galGal6;track=gc5BaseBw;maxItemsOutput=100 15. bigBed data from a UCSC database, chrom and start,end limits - api.genome.ucsc.edu/getData/track?genome=galGal6;track=ncbiRefSeqOther;chrom=chr1;start=750000;end=55700000 Search functions 1. Search matches within a UCSC Genome Browser genome assembly - api.genome.ucsc.edu/search?search=brca1&genome=hg38 2. Search matches within a UCSC Genome Browser genome assembly and restrict the search within the UCSC Genome Browser help documentation - api.genome.ucsc.edu/search?search=bigBed&genome=hg38&categories=helpDocs 3. Search matches within a UCSC Genome Browser genome assembly and restrict the search within the UCSC Genome Browser Public Hubs - api.genome.ucsc.edu/search?search=cerebellum&genome=hg38&categories=publicHubs 4. Search matches within a UCSC Genome Browser genome assembly and restrict the search within the track database (trackDb) settings - api.genome.ucsc.edu/search?search=signal&genome=hg38&categories=trackDb Error return examples 1. Request track data for non-existent chromosome in an assembly hub genome - api.genome.ucsc.edu/getData/track?hubUrl=http://hgdownload.soe.ucsc.edu/hubs/mouseStrains/hub.txt;genome=CAST_EiJ;track=assembly;chrom=chrI;start=43521;end=54321 2. Request track data from a restricted track. See FAQ - api.genome.ucsc.edu/getData/track?genome=hg19;track=decipherSnvs Practical examples Looking up the schema of a specific track The easiest way to get the schema for a track is to use the /list/schema function. This can be used on both file tracks and table tracks: Request schema for the hg38 CRISPR Targets track - api.genome.ucsc.edu/list/schema?genome=hg38;track=crisprAllTargets There is a second indirect way to get the schema which may be preferable in certain cases. When querying track data with the /getData/track function, the jsonOutputArrays can be used in conjunction to see the track schema. This includes a description of each field present in the track. The data will also be returned in JSON array type. Request data from hg38 gold track in array type - api.genome.ucsc.edu/getData/track?genome=hg38;track=gold;chrom=chrM;jsonOutputArrays=1 Hide track container information with trackLeavesOnly parameter When using the /list/tracks function to see the available tracks in an assembly, it can be useful to return all tracks in the same hierarchical level. By default, composite and supertracks will have the subtracks nested below, however, the trackLeavesOnly=1 parameter can be passed to hide the container information and display all tracks and subtracks at the same level. In the following example, the first link does not include the trackLeavesOnly parameter. The output can be compared to the second link to see the difference, which can be observed in the conservation track. In the first link, the multiz20way track is nested within the cons20way track. In the second link, however, the multiz20way subtrack is seen at an equivalent level with all other tracks, and the container, cons20way, is not present in the list. Request available tracks in the rn6 genome - api.genome.ucsc.edu/list/tracks?genome=rn6 Request available tracks in the rn6 genome, hiding container information - api.genome.ucsc.edu/list/tracks?genome=rn6;trackLeavesOnly=1 Requesting track data with over one million (1M) items in output Certain tracks may contain over 1M items. When these tracks are queried using the /getData/track function, only the first million items are returned. The API assumes this default value of 1M unless a different value (less than 1M) is specified with the parameter maxItemsOutput. One of these tracks is the knownGene track for hg19. Removing the maxItemsOutput parameter from the following link will lead to a 384Mb download, and may cause certain web browsers to time out. Request items in knownGene track of hg19, remove maxItemsOutput parameter for 1M max return - api.genome.ucsc.edu/getData/track?genome=hg19;track=knownGene;maxItemsOutput=5 There are different ways around this item limit, depending on how many items are in the track. For the knownGene track, breaking it down to component chromosome queries using the chrom parameter will suffice. In order to get a listing of the chrom names, and what chroms have data for that track, the /list/chromosomes function can be used. Request listing of chroms that have data for the knownGene track in hg19 - api.genome.ucsc.edu/list/chromosomes?genome=hg19;track=knownGene With the list of chrom names that have data, the /getData/track function can be used again while specifying the chrom parameter. In the following example, chr1 is queried and the itemsReturned field shows a total of 7967 items in the output, well below the 1M limit, meaning all data for chr1 has been extracted. This can then be repeated for all chroms of interest. Request items in knownGene track of hg19, only for chr1 - api.genome.ucsc.edu/getData/track?genome=hg19;track=knownGene;chrom=chr1 For tracks that have additional items, such as SNP tracks, the query can be further broken down using the additional start and end parameters. 
/goldenPath/help/haplotypes.html:Genome_Browser_Genome_Alleles	Gene Haplotype Alleles The Gene Haplotype Alleles feature displays the chromosome-phased 1000 Genomes Phase 1 data for protein coding regions. These data comprise the genomes of 1,092 individuals from 14 populations in Africa, Europe, East Asia and the Americas, constructed using a combination of low-coverage whole-genome and exome sequencing. The variant genotypes have been phased by the 1000 Genomes Project (i.e., the two alleles of each diploid genotype have been assigned to two haplotypes, one inherited from each parent). How to use the "Gene Haplotype Alleles" section Click on any protein-coding gene in the UCSC Genes track and scroll to the Common Gene Haplotype Alleles section. (The feature is currently implemented only on GRCh37/hg19 protein-coding genes.) There will be a table of haplotypes for the protein-coding portion of the gene. Each row in the table represents a unique gene haplotype as found in the 1000 Genomes Phase 1 project data. The table is sortable on any column by clicking on the column headers. Haplotype Frequency Haplotype frequencies are based upon all relevant chromosomes in the data set. The total number is almost always 2,184 (1,092 for Y and location-dependent for X). Hover over the frequency calculations to show the number of a particular haplotype in the dataset (e.g., "N=1327 of 2184"). If appropriate, the homozygous frequency will also be shown and will reflect the number of individuals in the dataset. The reference haplotype (made of entirely reference variants) may not be represented in the 1000 Genomes data. If it is, it will be so marked in the "Ref" column. Variant sites By default, only non-synonymous, common variant sites are displayed. Common variants are defined as occurring in at least 1% of 1000 Genomes subject chromosomes. includes rare and synonymous variant sites found in 1000 Genomes subjects in the list of haplotypes. limits the display to haplotypes defined by common and non-synonymous variation. displays variant sites (and full sequence) as DNA bases. displays variant sites (and full sequence) as predicted amino acids. Predicted stop codons are represented by "]" and predicted frameshifts by "[>>]". The reference variant is shown at the top of each variant site column. This is the value found in the GRCh37/hg19 reference genome at that variant site. In most cases it is a single letter (AA code/DNA base). In the case of an insertion with respect to the reference genome, the reference value is shown as "-". Large deletions are represented by the first two sequence letters followed by "+++". Hovering the pointer over any of the variant site links will show a more complete description of that variant. For example, the variant description "AA:16 A|T chr9:136137554 SNP: G|A (0.995|0.005) rs55917063" consists of the following elements: AA:16 A|T - AA residue number and variants (AA view only) chr9:136137554 - genome location SNP: G|A (0.995|0.005) - nucleotide variants and allele frequencies rs55917063 - dbSNP variant name (if one is known) Clicking on any non-reference variant shown in the variant sites columns will link to the full details of that variant site in the 1000 Genomes phase 1 track. Full sequence Each haplotype allele sequence is generated from GRCh37/hg19 reference DNA, with variants spliced in, then translated into amino acids. shows the predicted effects of variation on gene sequence for each of the haplotypes. If variant sites are currently displayed as DNA bases, then the predicted DNA sequence is shown (for coding regions only). If variant sites are displayed as amino acids, the predicted protein sequence is shown. simultaneously shows the DNA sequence above the protein sequence for easy comparison. Showing protein sequence with the DNA triplets is the easiest way to visualize the synonymous variants. shows the simplified protein sequences view. hides the full sequence view completely. Green vertical highlights accentuate the variant sites within the full sequence. Bold red letters mark the effects of variation. Synonymous changes are only evident when DNA bases are displayed. Blue vertical highlights show a variant that has been sorted on by clicking its column header. Sorting on a variant can be used to quickly locate one site out of many in the full sequence view. The AA residue number is shown when hovering over any part of the sequence in amino acid view. Rare haplotypes By default, only common gene haplotype alleles are displayed. Common haplotype alleles are defined as occurring in at least 1% of the relevant 1000 Genomes subject chromosomes. includes all haplotypes. Some large gene models cover many variants and therefore have a very large number of distinct haplotypes represented in the 1000 Genomes project data. If this is the case, only the 100 most frequently occurring haploptyes will be shown in the table, though the true number will be noted. limits the display to only common haplotypes. Populations Each haplotype is found in one or more subjects participating in phase 1 of the 1000 Genomes project. The 1000 Genomes populations, defined below, are grouped into broad categories (a.k.a. super populations). Haplotype distributions are available both for 1000 Genomes populations and for the major groupings. They are hidden by default. displays the distribution of the haplotypes across the major population groups. displays the distribution across the more specific 1000 Genomes groups. changes display from the 1000 Genomes grouping to the major grouping. hides the population columns. Each population group is shown as a column in the table, and each row shows the percent of that haplotype that is found in each group. This is not the same as the percent of each group that has the haplotype. Hover over the distribution numbers to show the frequency of occurrence of the haplotype within each group. For example, hovering over 25.7 might show "N=304 of 1183 (found in 71.0% of all ASN)", meaning that of the 1183 occurrences of the haplotype, 304 or 25.7% are found in the ASN group and that 71.0% of all East Asian copies of this gene (in 1000 Genomes phase 1 data) match this haplotype. To see the number of 1000 genomes chromosomes covered for each group, hover over the column header (e.g., ASN will usually show "East Asian [N=572]"). Scoring By default, scoring is hidden. Three types of scores are provided to help users find haplotype alleles that occur more or less frequently than expected or that have unusual distributions in populations. See definitions below. displays all score columns. hides all score columns. Population group definitions The numbers listed here are of individuals, but the numbers used in generating the haplotypes table are frequently the number of relevant chromosomes (e.g., 2184 not 1092). Major Groups Includes only major groups for which there are data in phase 1 of the 1000 Genomes project. AFR African 246 individuals AMR Ad Mixed American 181 individuals ASN East Asian 286 individuals EUR European 379 individuals 1000 Genomes Groups Includes only 1000 Genomes groups for which there are data in phase 1 of the project. African: ASW African Ancestry in Southwest US 61 individuals LWK Luhya in Webuye, Kenya 97 individuals YRI Yoruba in Ibadan, Nigeria 88 individuals Ad Mixed American: CLM Colombian in Medellin, Colombia 60 individuals MXL Mexican Ancestry in Los Angeles, California 66 individuals PUR Puerto Rican in Puerto Rico 55 individuals East Asian: CHB Han Chinese in Beijing, China 97 individuals CHS Han Chinese South 100 individuals JPT Japanese in Tokyo, Japan 89 individuals Europeans: CEU Utah residents with Northern and Western European ancestry 85 individuals FIN Finnish in Finland 93 individuals Gbr British in England and Scotland 89 individuals IBS Iberian populations in Spain 14 individuals TSI Toscani in Italia 98 individuals Scoring definitions Scores alone cannot be used to draw definitive conclusions about any haplotype. Hap score The haplotype score is based on the normalized (-log₁₀) probability of finding exactly N subject chromosomes with this haplotype, given the frequencies of individual variants and assuming they are independent. The score is normalized by multiplying the base probability by the total number of variants. Normalization allows comparing the scores between genes with many variant sites and those with few. The score will be positive if the haplotype is more frequent than expected by chance and negative if less frequent. Larger scores will result when minor variant alleles occur together more frequently than expected, which might reflect co-selection or may merely be an artifact of more recent events. A negative haplotype score may be more informative. For haplotypes made from common, non-synonymous variants, haplotype scores above 606 are seen in only 2% of genes. Likewise, a score of less than -199 is only seen in 2% of genes. Hom score The homozygous score is based upon the (-log₁₀) probability of finding exactly N individuals with this haplotype on both chromosomes, given the actual frequency of the haplotype in subject chromosomes. The score will be positive if the haplotype is found homozygous in more individuals than expected and negative when found in fewer than expected. Negative values might suggest that the haplotype is deleterious when homozygous. For haplotypes made from common, non-synonymous variants, homozygous scores above 92 are seen in only 2% of genes. Likewise, a score of less than -15 is only seen in 2% of genes. Pop score The population score (only visible when population distributions are displayed) is the fixation index (F_(ST)) based upon the difference in variance between sub-population haplotype frequencies and the total haplotype frequency. Note that this calculation is based upon the frequency of haplotype, rather than the distribution of that haplotype across populations. Nevertheless, large population scores should reflect large skews in distribution in more frequently occurring haplotypes. For haplotypes made from common, non-synonymous variants, population scores above 0.424 are seen in only 2% of genes and scores above 0.506 are seen in only 1% of genes. Notes: 1. If the gene is on the negative ('-' or "reverse") strand, all variant sites and sequences will be presented with respect to the negative strand. This differs from the way variants are displayed in the 1000 Genomes phase 1 variations track, which are always shown as they appear on the positive ('+' or "forward") strand. 2. Only variant sites occurring within coding exons are currently included in haplotypes. Variants occurring within intron splice junctions are not included. 3. Haplotypes are defined by the set of variant sites included and the variant allele at each of those sites. Therefore haplotype and homozygous frequency calculations depend upon which variant sites are included. Likewise all scores are specific to the haplotype as defined by the variant sites included, and population scores are also specific to the population groups that are examined. 4. The haplotypes displayed are not pregenerated but are derived from 1000 Genomes VCF files and other Genome Browser dataset at the time they are requested. Consequently, scoring is calculated in the context of a single gene model and the variant sites used to derive haplotypes. 5. If the number of variants covered exceeds 200, then the haplotype table will not be displayed and the reason so noted. 6. Certain viewing options are expensive operations which will slow the gene page response time. If this section is not being actively used, it is recommended that previous choices are cleared by pressing "Reset to defaults" at the bottom of this section. 
/goldenPath/help/hgCompositeHelp.html:Composite_Builder_Help	Composite Builder Help - DRAFT IN PROGRESS Contents Introduction Key Features Getting Started Resources and Support Introduction The Composite Builder is a tool which provides a quick and easy way to build a custom composite track grouping, without having to build a track hub. From a single interface, tracks that are being viewed in the UCSC Genome Browser display can be added into a new composite track group. Composite groups can then be customized with a suite of options (e.g., choosing colors for each sub-track, naming tracks, etc.). Your newly created composite group can then be viewed in the UCSC Genome Browser in reference to the particular genome assembly from which the custom composite was built. About composite tracks In the UCSC Genome Browser, a composite track is a grouping for a set of similar-type annotation tracks (e.g., all tracks store continuous bigWig data, or all tracks store data in the genePred format). A set of tracks within a composite track are referred to as "sub-tracks." Some of the natively-hosted annotation tracks within the UCSC Genome Browser are composite tracks. For example, the All GENCODE GRCh38/hg38 composite track is a group containing a set of GENCODE sub-tracks, where each track can have custom settings (such as different display modes per sub-track). In addition to the composite tracks hosted within the UCSC Genome Browser, track hubs also provide support for hub-based composite tracks. Creating a track hub is one of several ways to host your own customized data sets, which can be viewed within the UCSC Genome Browser environment (including the utilization of major tools such as the Table Browser). Key Features - Currently, the Composite Builder tool provides support for bigWig data only. - The Composite Builder builds groups per genome assembly. A group cannot contain tracks from more than one genome assembly. - Any track that is not set to "hide" in your current genome browser will appear as available to add into a new composite group. - Other... - Other.. - Other.. Getting Started The following section is a guide for getting started with the Composite Builder tool. By following the steps provided in the example below, you'll be able to create your own composite track, customize it, and view it with the UCSC Genome Browser. Step 1. Text about Step 1. [alt text 1] Step 2. Text about Step 2. [alt text 2] Step 3. Text about Step 3. Resources & Support - Questions? Please contact us. - Supported data types: Information about the bigWig format. - Documentation for composite tracks: Quick Start Guide to Hubs and the Hub Track Database Definition guide. - View the Composite Builder News Release - Credit to Dr. Bar-Joseph for assisting with wiggle sorting development: Bar-Joseph Z, Demaine ED, Gifford DK, Srebro N, Hamel AM, Jaakkola TS. K-ary clustering with optimal leaf ordering for gene expression data. Bioinformatics. 2003 Jun 12;19(9):1070-8. PMID: 12801867 
/goldenPath/help/bigWig.html:Genome_Browser_bigWig_Format	bigWig Track Format The bigWig format is useful for dense, continuous data that will be displayed in the Genome Browser as a graph. BigWig files are created from wiggle (wig) type files using the program wigToBigWig. Alternatively, bigWig files can be created from bedGraph files, using the program bedGraphToBigWig. bigWig also supports dynseq display as described in example 4. The bigWig files are in an indexed binary format. The main advantage of this format is that only those portions of the file needed to display a particular region are transferred to the Genome Browser server. Because of this, bigWig files have considerably faster display performance than regular wiggle files when working with large data sets. The bigWig file remains on your local web-accessible server (http, https or ftp), not on the UCSC server, and only the portion needed for the currently displayed chromosomal position is locally cached as a "sparse file". If you do not have access to a web-accessible server and need hosting space for your bigWig files, please see the Hosting section of the Track Hub Help documentation. Wiggle data must be continuous and consist of equally sized elements. If your data is sparse or contains elements of varying sizes, use the bedGraph format instead of the wiggle format. If you have a very large bedGraph data set, you can convert it to the bigWig format using the bedGraphToBigWig program. For details, see Example #3 below. Refer to this wiki page for help in selecting the graphing track data format most appropriate for the type of data you have. Note that the wigToBigWig utility uses a substantial amount of memory: approximately 50% more memory than that of the uncompressed wiggle input file. While running the wigToBigWig utility, we recommend that you monitor the system's memory usage with the top command. The bedGraphToBigWig utility uses about 25% more RAM than the uncompressed bedGraph input file. Creating a bigWig track To create a bigWig track from a wiggle file, follow these steps: Step 1. Create a wig format file following the directions here. When converting a wig file to a bigWig file, you are limited to one track of data in your input file; therefore, you must create a separate wig file for each data track. Step 2. Remove any existing "track" or "browser" lines from your wig file so that it contains only data. Step 3. Download the wigToBigWig program from the binary utilities directory. Step 4. Use the fetchChromSizes script from the same directory to create the chrom.sizes file for the UCSC database with which you are working (e.g., hg19). If the assembly genNom is hosted by UCSC, chrom.sizes can be a URL like: http://hgdownload.soe.ucsc.edu/goldenPath/genNom/bigZips/genNom.chrom.sizes Step 5. Use the wigToBigWig utility to create the bigWig file from your wig file: wigToBigWig input.wig chrom.sizes myBigWig.bw Note that the wigToBigWig program also accepts gzipped wig input files. Step 6. Move the newly created bigWig file (myBigWig.bw) to a web-accessible http, https, or ftp location. Step 7. If the file name ends with a .bigWig or .bw suffix, you can paste the URL directly into the custom track management page, click "submit" and view the file as a track in the Genome Browser. By default, the file name will be used to name the track. To configure the track label or other visualization options, you must create a track line, as shown in Step 8. Step 8. Construct a custom track using a single track line. The most basic version of the track line will look something like this: track type=bigWig name="My Big Wig" description="A Graph of Data from My Lab" bigDataUrl=http://myorg.edu/mylab/myBigWig.bw Paste the custom track line into the text box on the custom track management page. bigWig custom track lines can have several optional parameters, including: autoScale <on|off> # default is on alwaysZero <on|off> # default is off gridDefault <on|off> # default is off maxHeightPixels <max:default:min> # default is 128:128:11 graphType <bar|points> # default is bar viewLimits <lower:upper> # default is range found in data viewLimitsMax <lower:upper> # suggested bounds of viewLimits, but not enforced yLineMark <real-value> # default is 0.0 yLineOnOff <on|off> # default is off windowingFunction <mean+whiskers|maximum|mean|minimum> # default is maximum, mean+whiskers is recommended smoothingWindow <off|[2-16]> # default is off transformFunc <NONE|LOG> # default is NONE For further information on custom bigWig track settings, see the Track Database Definition Document. This includes information on how to add different colors to regions of the bigWig. For more information on how bigWig settings are used in native Genome Browser tracks, see the Configuring graph-based tracks page. Examples Example #1 In this example, you create a bigWig custom track using an existing bigWig file on the UCSC http server. The file contains data that spans chromosome 21 on the hg19 assembly. To create a custom track using this bigWig file: 1. Paste the URL http://genome.ucsc.edu/goldenPath/help/examples/bigWigExample.bw onto the custom track management page for the human assembly hg19 (Feb. 2009). 2. Click the "submit" button. 3. On the next page that displays, click the "chr21" link in the custom track listing to view the bigWig track at position chr21:33,031,597-33,041,570 in the Genome Browser. Alternatively, you can customize the track display by including track and browser lines that define certain parameters: 1. Construct a track line that references the bigWigExample.bw file: track type=bigWig name="Example One" description="A bigWig file" bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigWigExample.bw 2. Include the following browser line to ensure that the custom track opens at the correct position: browser position chr21:33,031,597-33,041,570 3. Paste the browser and track lines onto the custom track management page for the human assembly hg19 (Feb. 2009), click the "submit" button, then click the "chr21" link in the custom track listing to view the bigWig track in the Genome Browser. Example #2 In this example, you will create your own bigWig file from an existing wiggle file. 1. Save this wiggle file to your computer (Steps 1 and 2 in Creating a bigWig format track, above). 2. Save the file hg19.chrom.sizes to your computer. This file contains the chrom.sizes for the human (hg19) assembly (Step 4, above). 3. Download the wigToBigWig utility (step 3, above). 4. Run the utility to create the bigWig output file (step 5, above): wigToBigWig wigVarStepExample.gz hg19.chrom.sizes myBigWig.bw Place the newly created bigWig file (myBigWig.bw) on a web-accessible server (step 6, above). Paste the URL of the bigWig file into the custom track entry form, or construct a track line that points to your bigWig file (step 7, above). Create the custom track on the human assembly hg19 (Feb. 2009), and view it in the Genome Browser (step 8, above). Note that the original wiggle file spans only chromosome 21. Example #3 To create a bigWig track from a bedGraph file, follow these steps: Create a bedGraph format file following the directions here. When converting a bedGraph file to a bigWig file, you are limited to one track of data in your input file; therefore, you must create a separate bedGraph file for each data track. Remove any existing track or browser lines from your bedGraph file so that it contains only data. Download the bedGraphToBigWig program from the binary utilities directory. Use the fetchChromSizes script from the same directory to create the chrom.sizes file for the UCSC database with which you are working (e.g., hg19). If the assembly genNom is hosted by UCSC, chrom.sizes can be a URL like http://hgdownload.soe.ucsc.edu/goldenPath/genNom/bigZips/genNom.chrom.sizes Use the bedGraphToBigWig utility to create a bigWig file from your bedGraph file: bedGraphToBigWig in.bedGraph chrom.sizes myBigWig.bw (Note that the bedGraphToBigWig program DOES NOT accept gzipped bedGraph input files.) 5. Move the newly created bigWig file (myBigWig.bw) to a web-accessible http, https, or ftp location. 6. Paste the URL into the custom track entry form or construct a custom track using a single track line. 7. Paste the custom track line into the text box on the custom track management page. Example #4 In this example, we will display a bigWig with a dynamic sequence logo (Motif Logo) using logo=on. 1. Construct a track line that references the bigWigExample.bw file: track type=bigWig logo=on name="Example dynseq" description="A bigWig file with logo=on dynseq" visibility=full autoScale=off bigDataUrl=http://genome.ucsc.edu/goldenPath/help/examples/bigWigExample.bw 2. Include the following browser line to ensure that the custom track opens at the correct position: browser position chr21:33,037,000-33,037,050 3. Paste the browser and track lines onto the custom track management page for the human assembly hg19 (Feb. 2009), click the "submit" button, then click the "chr21" link in the custom track listing to view the bigWig track in the Genome Browser. You can also load the above by clicking this link. [Example4 of dynseq logo=on setting] This dynseq display scales nucleotide characters by user-specified, base-resolution scores and was developed by the Kundaje Lab. Sharing your data with others If you would like to share your bigWig data track with a colleague, learn how to create a URL by looking at Example #6 on this page. Extracting data from the bigWig format Because bigWig files are indexed binary files, it can be difficult to extract data from them. UCSC has developed the following programs to assist in working with these files, available from the binary utilities directory. - bigWigToBedGraph — converts a bigWig file to ASCII bedGraph format. - bigWigToWig — converts a bigWig file to wig format. Note: if a bigWig file was created from a bedGraph, bigWigToWig will revert the file back to bedGraph. - bigWigSummary — extracts summary information from a bigWig file. - bigWigAverageOverBed — computes the average score of a bigWig over each bed, which may have introns. - bigWigInfo — prints out information about a bigWig file. These utilities accept either file path names or URLs to files as input. As with all UCSC Genome Browser programs, simply type the program name (with no parameters) on the command line to view the usage statement. In some cases, bigWigSummary and bigWigAverageOverBed will produce very similar results, but in other cases, the results may differ. This is due to data-handling differences between the two programs. Summary levels are used with bigWigSummary; therefore, some rounding errors and border conditions are encountered when extracting data over relatively small regions. In contrast, the bigWigAverageOverBed utility uses the actual data, which ensures the highest level of accuracy. 
/goldenPath/help/publicHubGuidelines.html:Public_Hub_Guidelines	Public Hub Guidelines The Genome Browser provides links to a collection of public hubs that have been registered with UCSC and are available to view on the Public Hubs page. Here are guidelines for those who are trying to make a hub a UCSC public hub. If you have created a hub that meets the requirements and is of general interest to the research community, please contact us at genome-www@soe.ucsc.edu to have it added to the list. As a reference for interpreting trackDb.txt settings, use the Hub Track Database Definition glossary. For information on using the Track Hub features, refer to the Genome Browser Track Hub User Guide. See also the Basic Hub Quick Start Guide, Quick Start Guide to Organizing Track Hubs into Groupings, Track hub settings blog post, Quick Start Guide to Assembly Hubs and Quick Start Guide to Searchable Track Hubs. Contents Required Guidelines Recommended Guidelines Public Hub Examples Required Guidelines The following guidelines must be met before your hub will be added to our public list: Required for both track and assembly hubs: - You MUST have a description page for every configuration page (composite, superTrack or stand alone track). Note that multiple tracks and/or composites can use the same description page with the "html" setting. You can find more information on creating track description pages in the recommendations section below. - All of your description pages MUST have a contact email address prominently displayed. - At least one track should have a visibility set to display (in full, pack, squish, or dense), and try to have no more than 10 tracks enabled by default upon first connecting your hub. - Have a descriptionUrl html page specified in your hub.txt. This should be a URL to a description page for your entire hub, often public hubs will link to a full-text paper or to their laboratory webpage that describes the research presented in the hub. These links are presented on the Public Hubs page as a hyperlink on the longLabel presented in the hub.txt, while the shortLabel is a hyperlink to the hub.txt location. Required for only assembly hubs: Add a gateway page for each assembly by having a htmlPath line for each genome not already hosted by UCSC in the genomes.txt. The following settings should properly be set in your genomes.txt (The last 3 settings will make it easier to find assembly hub species in hgGateway by UI search): - defaultPos - scientificName - organism - description Recommended Guidelines These guidelines in the following sections are recommended to improve user experience, but are not required to be implemented before the hub is added to our list of Public Hubs. Note on stability Keep in mind that users may start to rely on your track hub for their work. If the track hub web server is down or the URL changes, users of the track hub will have no access to the data. Users may also have stable session links in manuscripts that include the track hub data and the sessions could all stop working. We check public track hubs periodically and send an email after a 24-hour downtime. We will remove track hubs if they are offline for several days. Contact us (genome-www@soe.ucsc.edu) if there is a change such as moving webservers of the track hub. Sudden changes can also impact users where large changes to the track hub can change the analysis of users such as removing tracks or changing options. In these cases, keeping a previous version of the tracks and making them in a different track group with suffixes such as "V1", "(previous versions)" or hint in the track long labels. Labeling tracks with informative labels will help users. You can also add a "dataVersion" trackDb statement to indicate to users what version of the data is being used. Track organization recommendations Related tracks can be grouped in a few different ways, namely superTracks, multiWigs, and composites. If your hub includes a large number of tracks, the grouping of tracks may be necessary. This will prevent your hub's track group from being an overwhelming mess of individual tracks and can make user configuration of your tracks easier. Composite tracks Related tracks of the same data type (e.g. a set of related bigBed tracks) should be combined into composites where appropriate. Have multi-view only when there is more than one view. Views ideally give alternate access to the same data (e.g. signals and called peaks). Keep in mind that the value of views is that they allow for more than one data/configuration type (e.g. bigBed and bigWig) in a single composite. All subtracks of a view must have the same data type. Likewise, all subtracks of a non-multi-view composite must be the same type. Recommendations for using dimensions with your composite tracks: - There should be no dimensions with a single entry (do not have only one cell line represented in dimX=cell), unless data growth is expected to fill in additional entries. - Using only one dimension: preferably use dimX (e.g. dimensions dimX=cell). This saves vertical User Interface space, but is not always the best choice. - Using two dimensions: use dimX and dimY (e.g. dimensions dimX=cell dimY=mark) - Using more than two: use dimX, dimY on the most important dimensions. Then use dimA,B,C as needed on lesser dimensions. (e.g. dimensions dimX=cell dimY=mark dimA=donor_id) - The A,B,C dimensions should probably use filterComposite (e.g. filterComposite dimA) - Each dimension and views should be represented in sortOrder, ideally in order of dimX, dimY, dimA,B,C, view (e.g. sortOrder cell_type=+ mark=+ donor_id=+ view=+). - Tags of subGroup/dimension should be short and sweet with no special chars. Also labels can have HTML codes embedded (e.g. NOT CPG_methylation_%=CPG_methylation_% RATHER mpct=CPG_methylation_&_#37) - Never represent the same subgroup in both view and as a dimension (e.g. NOT dimensions dimX=view). A subgroup should never be in two dimensions (e.g. NOT dimensions dimX=cell dimY=mark dimA=cell). The composite will appear to function but multiple ways of selecting the same thing will create a confusing and inconsistent user interface. Super tracks Extremely large hubs may use superTracks as well to achieve a meaningful hierarchy. Super tracks can be used to group together any type of related tracks; for example, you could combine a multiWig, a composite, and a bigBed track together into a single superTrack. Track display recommendations - Avoid setting a composite track and all of its subtracks to the same visibility. When you have composite tracks that are hidden by default, it is best to still designate some subtracks to display when the composite track is turned on (visibility dense, versus the default of hide). This provides an example of your track data to users who turn on your composite track. If no subtracks are turned on by default, a user who changes your composite track visibility to "show" won't see anything. - The shortLabel text should be under 20 characters, or meaningful information may be cut off from display when tracks are set to "dense" visibility. Track description page recommendations - The description page should preferably contain UCSC's standard track description, Display Conventions and Configuration, Methods, Credits, and References. More information can be found on the template page. - Your track description pages should provide meaningful documentation for your tracks. - If you are creating a hub based on a paper, use the paper's abstract as a starting point for your track's description section - The Methods section expand upon the overview of the Description section and provide more details about how the data for the track was produced - You should assume a broad audience of students and researchers will use your hubs. You should spell out common acronyms for those who may be new to genomics. For example, you might write out a term and its acronym as follows "Fluorescent in situ hybridization (FISH)" which spells it out and then provides the acronym that you can use throughout the rest of your description page. - It might be a good idea to include a "Data Access" section on your track description page which describes how to access the data in your hub and where to download the raw data for the tracks in your hub. Public Hub Examples Many of the public hubs in the Genome Browser provide excellent examples or templates for creating your own hub. As a reference for interpreting trackDb.txt lines used in these example hubs, please refer to the Hub Track Database Definition glossary. Some Hub Track Database Definition settings like filters have additional help documentation. Also note that if you are only displaying one genome you can use the useOneFile on setting. Example Track Hubs Example 1 The Principal Splice Isoforms APPRIS hub provides a good example of basic hub that includes a few different annotation tracks. Each track includes its own description page and is colored in such a way that distinguishes it from the other tracks in the hub and native track in the UCSC Genome Browser. Here are some links to their configuration files and some description pages: - hub.txt - genomes.txt - trackDb.txt for the default hub assembly, hg38 - Description page for APPRIS - Principal Isoforms track - The track description on the human GRCh38/hg38 Genome Browser Example 2 The Roadmap Epigenomics Integrative Analysis Hub provides a great example of how you might use organize your hub if you have thousands of different tracks. The hub uses composites with dimensions to organize thousands of different tracks across a number of cell lines and uses supertracks to group these tracks even further. Here are some links to their configuration files and some description pages: - hub.txt named "RoadmapIntegrative.txt" - genomes.txt named "roadmapintegrativeall.txt" - trackDb.txt named "roadmap_both_02182015_trackDb.txt" for hg19 - The track description on the human GRCh37/hg19 Genome Browser Example Assembly Hub The C elegans isolates hub provides an excellent example of what your assembly hub could look like. The hub creators provide a detailed description page for each assembly, many different annotations tracks each with their own description page, and clearly defined track groups with those related tracks grouped together. Here are some links to their configuration files and some description pages: - hub.txt - genomes.txt - trackDb.txt for the primary genome in the hub, CB4856Princeton_JR-contig - groups.txt that defines track groups for CB4856Princeton_JR-contig - Description page for CB4856Princeton_JR-contig genome - Gateway page - The description page for Rajewsky Mixed Stage RNAseq. The track description on the Genome Browser - The description page for WS230 cDNA blat Annotations. The track description on the Genome Browser 
/goldenPath/help/codonPhase.html:Genome_Browser_Codon_Phase_Display	Codon phases shown on the Genome Browser Gene transcript tracks, for example NCBI RefSeq and Gencode, show the codon phase of exons on mouse over. The text looks like this: Transcript: ENST00000297261.7 Strand: - Exon: Exon 2 of 3 Codon phase: start 0, end 1: out-of-frame exon The codon phase specifies where in the coding sequence the intron falls in the codon that separates two exons: - Phase 0: before the first nucleotide, cleanly between two codons. - Phase 1: between the first and the second nucleotide - Phase 2: between the second and third nucleotide This means that: - The start phase of an exon is always the same as the end phase of the previous exon. - An exon where the end and start phases are identical can be removed without destroying the reading frame: in-frame exon. - An exon where start and end phases differ cannot be removed without destroying the reading frame: out-of-frame exon. 
/goldenPath/help/net.html:Genome_Browser_Net_Format	Net Format The net file format is used to describe the axtNet data that underlie the net alignment annotations in the Genome Browser. For a detailed description of the methods used to generate these data, refer to the Genome Browser description pages that accompany the downloadable net alignment tracks. You can find a detailed example how nets are constructed on our wiki. At the beginning of each target species chromosome, a "net" line appears with the format: net chromName chromSize chromName is the target species chromosome name, and chromSize is the chromosome size. For example: net chr2L 23011544 Each target chromosome section in the file starts with a net line. The net line is followed by a set of "fill" and "gap" lines. File indentation: Line indentation level represents the parent/child relationship between records and is a necessary part of the net file format. Child records are indented one space from the parent, as shown in the example net file below. net chr2L 23011544 fill 6004 3278 chrXR_group3a - 1396397 2164 id 25606 score 23114 ali 782 qDup 576 type top tN 0 qN 0 tR 36 qR 0 tTrf 0 qTrf 0 gap 6065 2 chrXR_group3a - 1398498 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 6096 1485 chrXR_group3a - 1397572 897 tN 0 qN 0 tR 36 qR 0 tTrf 0 qTrf 0 fill 6096 513 chrU - 5570675 533 id 48675 score 4435 ali 465 qDup 533 type nonSyn tN 0 qN 0 tR 0 qR 13 tTrf 0 qTrf 0 gap 6116 8 chrU - 5571188 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 6156 5 chrU - 5571156 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 6184 3 chrU - 5571133 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 6212 18 chrU - 5571106 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 6244 9 chrU - 5571092 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 6340 2 chrU - 5570996 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 6515 3 chrU - 5570771 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 7623 1 chrXR_group3a - 1397530 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 7664 1007 chrXR_group3a - 1397008 482 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 fill 7664 382 chrXL_group1e - 8262003 506 id 25608 score 10609 ali 364 qDup 506 type nonSyn tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 7784 4 chrXL_group1e - 8262361 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 7792 3 chrXL_group1e - 8262357 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 7921 2 chrXL_group1e - 8262126 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 7949 9 chrXL_group1e - 8262092 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 8693 1 chrXR_group3a - 1396985 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 fill 9833 1251 chrU - 5562980 1239 id 48675 score 10720 ali 1124 qDup 1094 type top tN 0 qN 0 tR 22 qR 88 tTrf 0 qTrf 0 gap 9966 7 chrU - 5564075 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 10015 3 chrU - 5564030 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 10088 2 chrU - 5563957 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 gap 10101 8 chrU - 5563946 0 tN 0 qN 0 tR 0 qR 0 tTrf 0 qTrf 0 Field definitions The net file consists of 7 fixed fields and a set of optional name/value pair fields. In the descriptions below, target refers to the reference species and query refers to the aligning species. Fixed fields - Class -- Either fill or gap. Fill refers to a portion of a chain - Start in chromosome -- (target species) - Size -- target species) - Chromosome name -- (query species) - Relative orientation -- between target and query species - Start in chromosome -- (query species) - Size -- (query species) Optional fields (Name/value pairs) - id -- ID of associated chain (gapped alignment), if any. - score -- Score of associated chain. - ali -- Number of bases in alignments in chain. - qFar -- For fill that is on the same chromosome as parent, how far fill is from position predicted by parent. This helps determine if a rearrangement is local or if a duplication is tandem. - qOver -- Number of bases overlapping with parent gap on query side. Generally, this will be near zero, except for inverts. - qDup -- Number of bases in query region that are used twice or more in net. This helps distinguish between a rearrangement and a duplication. - type -- One of the following values: - top -- Chain is top-level, not a gap filler. - syn -- Chain is on same chromosome and in same direction as parent. - inv -- Chain is on same chromosome on opposite direction from parent. - nonSyn -- Chain is on a different chromosome from parent. - tN -- Number of unsequenced bases (Ns) on target side. - qN -- Number of unsequenced bases on query side. - tR -- Number of bases in RepeatMasker masked repeats on target. - qR -- Number of bases in RepeatMasker masked repeats on query. - tNewR -- Bases in lineage-specific repeats on target. - qNewR -- Bases in lineage-specific repeats on query. - tOldR -- Bases in repeats predating split on target. - qOldR -- Bases in repeats predating split on query. - tTrf -- Bases in trf (Tandem Repeat Finder) repeats on target. - qTrf -- Bases in trf repeats on query. 
/goldenPath/help/trackDbIndexBb.html:trackDbIndexBb_|_UCSC_Genome_Browser	trackDbIndexBb - A utility to support hideEmptySubtracks on Composite Tracks When there are many subtracks in a composite view, it may be useful to limit the display to only those with data in the current viewing window. The trackDb setting, hideEmptySubtracks, enables this behavior. This track setting produces a checkbox on the track configuration page allowing the user to enable or disable this feature. If it is configured to 'on', then the feature will be on by default (the checkbox is checked). To take full advantage of this setting it is helpful, though not always required, to index the underlying bigBed files, using the trackDbIndexBb utility. This utility creates multibed/index files containing the coordinates where the tracks intersect, expediting data lookup. There are two instances in which these files are helpful or required: - (helpful) In large composites (dozens or hundreds of tracks), especially when subtrack data are sparse, the index files will provide a substantial performance improvement. - (required) For building track associations. An example of this is when it is desired to display peak and signal tracks together. Because hideEmptySubtracks works only on bigBed tracks (peaks), associated tracks (such as bigWig signals) can be designated to be displayed alongside the bigBeds with data. To build the index files, first download the trackDbIndexBb utility. For more information on downloading our command line utilities, see these instructions. There are three other programs needed to run trackDbIndexBb. Two of them, bedToBigBed and bigBedToBed, can be found in the same downloads directory. The final dependency, bedtools, can be found on the bedtools site. Parameters Kent utilities can be run with no parameters to display a usage message. Additionally, trackDbIndexBb can be passed the - h flag to display a more verbose help message. ./trackDbIndexBb ./trackDbIndexBb -h Below is a short description of the parameters: - trackName | The name of the composite containing the bigBed tracks to be indexed. Higher-level composite names may be used to build track associations. This means that all tracks in the hierarchy below trackName will be searched for indexing and associations. - raFile | The location of the trackDb.ra file containing the bigDataUrls of the bigBeds to be indexed. - chromSizes | Location of the chrom.sizes file. This file contains the names and sizes of the chromosomes for the working assembly. This can be generated from the 2bit genome, downloaded from the respective assembly on our download server, or fetched using the fetchChromSizes utility found in the same directory as trackDbIndexBb. - -o --out | (Optional) Path to the output directory where the resulting files will be placed. Defaults to current directory. - -p --pathTools | (Optional) Location where the dependent programs can be found. Will automatically check current directory and user's path. trackDbIndexBb has three dependencies listed above. Note that bedtools must be downloaded from an external group. - -n --noDelete | (Optional) Keep intermediary multiBed file. - -m --metaDataVar | (Optional) Used only when building track associations. The variable designated here will be a trackDb variable which can be used to associate tracks. Default value is subGroups, though metaData is also commonly used. See example below for more information. - -s --subGroupRemove | (Optional) Used only when building track associations. In conjunction with --metaDataVar, this variable is used to build track associations. The value designated will be excluded as a matching requirement from the trackDb parameter. By default, view is used, though this can change depending on data and organization. See example below for more information. Example 1 - Building Index Files In this first example, we will build index files for a composite track containing 12 bigBed files. Index files hardly improve performance on a track with so few files, however, the steps would be the same on a larger track. First, we can take a look at the header stanza for the composite. The complete trackDb.ra file is available here. track problematic shortLabel Problematic Regions longLabel Problematic Regions for NGS or Sanger sequencing or very variable regions compositeTrack on hideEmptySubtracks off group map visibility hide type bigBed 3 + We can see that the hideEmptySubtracks setting is already enabled, set off by default. The index files we are building are not required, but instead, improve the performance of the feature. This also gives us the composite track name, problematic. We will want to pass this as our trackName variable. The other two required parameters are the path to the trackDb.ra file, and the chrom.sizes file. If we assume both of those are in the current directory, and that the required dependencies are present in the path, we can run trackDbIndexBb as such: ./trackDbIndexBb problematic smallExampleTrackDb.ra hg19.chrom.sizes This will generate two files in the current directory: problematic.multiBed.bb problematic.multiBedSources.tab We can then enable the use of these index files for hideEmptySubtracks by adding the following two lines to our trackDb.ra file, adjusting the path if needed: hideEmptySubtracks off hideEmptySubtracksMultiBedUrl problematic.multiBed.bb hideEmptySubtracksSourcesUrl problematic.multiBedSources.tab Example 2 - Creating Track Associations In this longer example, we are looking to build index files with track associations between DNase-seq peak and signal tracks. There are 2 bigBed peak tracks and 4 bigWig signal tracks. The complete trackDb for the example can be found here. Looking at the top level stanza, we see that this composite track has two views, one for peaks and one for signals. The data are associated with a few different subGroups: track uniformDnase subGroup4 lab Lab Duke=Duke UW=UW UWDuke=UW-Duke subGroup3 view View Peaks=Peaks Signal=Signal subGroup2 cellType Cell_Line GM12878=GM12878 H1-hESC=H1-hESC To help us decide how to best make these associations, let us see what parts of the peak and signal stanzas we would like to associate are relevant: track wgEncodeUWDukeDnaseGM12878FdrPeaks type bigBed 6 + parent uniformDnasePeaks on bigDataUrl wgEncodeUWDukeDnaseGM12878.fdr01peaks.hg19.bb subGroups view=Peaks tier=t1 cellType=GM12878 lab=UWDuke metadata cell=GM12878 track wgEncodeDukeDnaseGM12878FdrSignal type bigWig parent uniformDnaseSignal on bigDataUrl wgEncodeOpenChromDnaseGm12878Aln_5Reps.norm5.rawsignal.bw subGroups view=Signal tier=t1 cellType=GM12878 lab=Duke metadata cell=GM12878 lab=Duke track wgEncodeUWDnaseGM12878FdrSignal type bigWig parent uniformDnaseSignal on bigDataUrl wgEncodeUwDnaseGm12878Aln_2Reps.norm5.rawsignal.bw subGroups view=Signal tier=t1 cellType=GM12878 lab=UW metadata cell=GM12878 lab=UW The first track is the bigBed peaks track (peaks view) and the second and third are bigWig signal tracks (signal view). hideEmptySubtracks allows for two optional variables to build track associations. The first, -m --metaDataVar, designates which trackDb variable will be used to build the association. In this example, the peaks are determined using a combination of the signal tracks, therefore, we would like to display both of the signal tracks whenever the peak track has data. At this point, it is important to explain how trackDbIndexBb makes track associations. It will look at the stanza variable line designated by -m --metaDataVar, then look for identical matching lines in other stanzas. Since at least one parameter within will usually differ, such as the designation between peak and signal, -s --subGroupRemove can be used to strip out one of the parameters in the line. The subGroups parameter could be used. However, we see that the two variables that differ between the peak and signal stanzas are view and lab. We would have to strip both of those to have matching parameter variables and build an association. Alternatively, we could use the metaData parameter. This parameter associates the tracks by the cell, with only the lab variable differing. This would be the best choice as only a single parameter would have to be stripped, lab, as opposed to two, lab and view, to have matching peak and signal parameters for related tracks. Now that we know which parameter we would like to use to build associations, we need to use the second optional parameter, -s --subGroupRemove, to tell hideEmptySubtracks which variables to strip out in making the association. In this case, we would like to keep the cell variable, but strip the lab. This means that lab will be the parameter passed. In this way, associations will be made between any tracks that match the contents of their metaData parameter once the lab variable has been stripped out. Now that we have chosen our parameters, we will run the utility -- assuming our chrom.sizes file, our trackDb.ra file, and all the supporting programs (bedToBigBed, bigBedToBed, bedtools) are present in the current directory. We will also choose the output to be the current directory: ./trackDbIndexBb uniformDnase exampleTrackDb.ra chrom.sizes -o . -p . -m metadata -s lab Note that in this case, we could have omitted the -o and -p values as the current directory is the default for both. In this small example, the utility would run in a few seconds. But larger inputs containing hundreds of tracks can take hours. Upon completion, two files will be generated: uniformDnase.multiBed.bb uniformDnase.multiBedSources.tab The .bb file will be a big multibed containing the coordinates where the tracks intersect, expediting data lookup, and the .tab file will serve as an index for the multibed while also containing the track associations. The .tab file can be quickly examined to ensure proper generation as it should contain a numerical first column, followed by the bigBed track, then any number of desired track associations, e.g. 1 wgEncodeUWDukeDnaseGM12878FdrPeaks wgEncodeDukeDnaseGM12878FdrSignal wgEncodeUWDnaseGM12878FdrSignal 2 wgEncodeUWDukeDnaseH1hESCFdrPeaks wgEncodeDukeDnaseH1hESCFdrSignal wgEncodeUWDnaseH1hESCFdrSignal Finally, hideEmptySubtracks can be enabled and pointed to the newly generated files on the top composite stanza: hideEmptySubtracks on hideEmptySubtracksMultiBedUrl uniformDnase.multiBed.bb hideEmptySubtracksSourcesUrl uniformDnase.multiBedSources.tab More information on how to use track hubs can be found in the Track Hub help page as well as the Track Database Definition Document. 
/goldenPath/help/hgCnvColoring.html:Genome_Browser_Codon_Coloring	Structural Variants Color Code CNV items in the Genome Browser tracks are colored by Variant Type. Three levels of color intensity are used to indicate the Clinical Significance. The lightest shade indicates benign/likely-benign variants, the darkest shade indicates pathogenic/likely pathogenic variants, and the medium shade indicates other or no clinical significance has been defined. Variant Type Benign Other Pathogenic ---------------------------- -------- ------- ------------ Gain Loss Insertion Inversion Structural Alteration Other Sequence Alterations Each color category includes various Variant Types: - Gain: duplications (SO:1000035), tandem duplications (SO:1000173), and copy number gains (SO:0001742) - Loss: deletions (SO: 0000159), and copy number losses (SO:0001743) - Insertion: insertions (SO: 0000667), indels (SO: 1000032), and mobile element insertions (SO: 0001837) - Inversion: inversions (SO: 1000036) - Structural Alteration: translocations (SO: 0000199), complex structural alterations (SO: 0001784), complex chromosomal rearrangements (SO: 0002062), fusions (SO: 0000806), and multi-allele CNVs - Other Sequence Alterations: short tandem repeat expansions (SO: 0002162), short tandem repeat contractions (SO: 0002163), microsatellites (SO: 0000289), and undetermined or undefined alterations 
/goldenPath/help/hgIntegratorHelp.html:Genome_Browser_FAQ	Data Integrator User's Guide Contents Introduction Video demo Getting started: performing a simple query Selecting data sources Computing item overlap Configuring output ------------------------------------------------------------------------ Search the Genome Browser help pages:   Questions and feedback are welcome. Introduction The Data Integrator is a fast and powerful graphical interface that can combine and export data from multiple tracks simultaneously. Like the Genome Browser and Table Browser, it can combine data from the browser database, user custom tracks and track hubs. Using the Data Integrator, you can retrieve tab-separated text for the data underlying up to five tracks, combining multiple tracks' data for items whose genomic positions overlap the positions of items in the first selected track. Depending on your needs, you may: - include all columns of all data sources in your query, or only the columns that you select. - query genome-wide (provided that the results are not so large that the connection times out), or only for your regions of interest. - view results in your web browser window, or download to a local file, optionally compressed by gzip. The Data Integrator does not yet have many of the features of the Table Browser tool. For example, if you want to retrieve DNA sequence or GTF format, you will need to use the Table Browser. Video demonstration of the Data Integrator []       Visit our Video Page.       Visit our YouTube channel. Getting started: performing a simple query Follow these instructions to perform a basic query that intersects two or more tracks: Step 1. Pick a genome assembly Specify the genome assembly from which you'd like to retrieve the data by choosing the appropriate organism using the group and genome menus, then selecting the assembly version from the assembly menu. Note that changing the group menu causes the genome menu to refresh, and changing the genome menu causes the assembly menu to refresh. Assemblies that are uploaded as part of an assembly hub will be available under the hub's name in the group menu. Step 2. Pick a genomic region The Data Integrator allows you to choose the genomic regions for which you want output. - To display all of the data records for the selected track(s), select genome from the region to annotate menu (not available for certain tracks with restrictions on data sharing). - To restrict the data to a specific position range, select position or search term from the region to annotate menu and type or paste the position into the box. Some examples of specific positions include a chromosome name (chrX), a coordinate range within a chromosome (chrX:100000-400000), or a scaffold name. - To look up the position range of a genomic element -- such as a gene name, an accession ID, an STS marker, etc. -- or keywords from the GenBank description of an mRNA, select position or search term from the region to annotate menu and then type your search term into the box. For assemblies that have gene annotations, autocompletions of gene symbols are offered. If your search term matches more than one track or item, then a pop-up will appear with matching items and positions; click an item to select its position. - To query multiple genomic regions, select define regions from the region to annotate menu and then enter up to 1,000 regions in a 3- or 4-field BED file format or as chrom:start-end position ranges. Step 3. Configure your data sources The track group menu shows all the annotation track groups available in the selected genome assembly. The names correspond to the groupings displayed at the bottom of the Genome Browser annotation tracks page. When a group is selected from the menu, the track menu automatically updates to show all the annotation tracks available within that group. - To examine all the tracks available within a certain group (e.g., all gene prediction tracks), select the group name from the track group menu, then browse the entries in the track menu. - Custom annotation tracks created during the current session are listed under the Custom Tracks group. - Each connected track hub will be listed as its own group under the track group menu. - Clicking the "View table schema" link opens a new web browser tab with the Table Browser's schema page which describes the data fields. [hgIntegrator add track] After selecting your track from the drop-down menus, click Add. You may add up to five tracks. Some tracks are not available when the region is set to genome due to the data provider's restrictions on sharing. See the Selecting Data Sources section for more details. Step 4. Rearrange your data sources (Optional) You can rearrange your data sources by clicking on a track name and dragging it into the desired order. Items are output based on their overlap with the first listed track. The order of the tracks will also determine the order of the fields in the output. Fields from tracks higher in the list will appear first in the output. Step 5. Choose output fields (Optional) Click Choose fields... to bring up a checklist of all fields from all selected data sources. By default, every field is selected. You can choose the columns you want in your output by checking the box next to that field. Additionally, you can add and select fields from up to four related tables. See the Configuring Output section for more details. Step 6. Get output Click Get output to display the results of the query in the web browser window. The results will be displayed as tab-delimited text. Download results as a file by checking the box next to Send output to file and entering a file name into the box. The output file can also be compressed by gzip. Selecting data sources The Data Integrator allows you to select up to five data sources to intersect. These data sources can be tracks provided by UCSC or your own data uploaded using custom tracks or track hubs. Only positional tables, such as BED, wiggle, and genePred, are currently allowed in the output. Other formats such as MAF or BAM are currently unavailable in the Data Integrator. Items in your selected tracks are output based on their overlap with items in your primary track. The primary track is the one that shows up first in the list of your chosen data sources. In the following example, UCSC Genes would be the primary track and items would be output based on their overlap with items in UCSC Genes: [hgIntegrator configure data Sources] The order of your selected tracks also affects the order of fields in your output. Fields from tracks that are higher up in the Configure Data Sources box will show up before tracks that are lower in that box. Using the same order of tracks seen in the previous example, this means that fields from the UCSC Genes track will appear first in the output, then fields from DNase Clusters and lastly fields from Conservation - 100 Vert. El. However, your tracks aren't limited to the order in which you selected them, and you can rearrange the order of your tracks at any time. To do so, just click and drag a track to a new position in the list. A track can be removed by clicking the "X" next to it. Some data providers stipulate that their data can be browsed visually in the Genome Browser, but cannot be provided for download via the Table Browser or Data Integrator. Tracks from those providers do not appear in the track menus. Other providers allow queries within specific regions, but not genome-wide queries. Their tracks appear in the menus, but are grayed out when region is set to genome. Computing item overlap The current intersection mechanism the Data Integrator uses is fairly simple. For gene transcripts, mRNAs, and any other tracks where the items have blocks or gaps, the overlap for the entire range of the item is computed. This means that introns and gaps in the item are also considered when computing overlap. This is an improvement upon the Table Browser which only computed overlap with exons. For example, let's consider the following situation using the UCSC Genes and DNase Clusters tracks: [hgIntegrator hgTracks example] If we were using the Data Integrator to find the DNase peaks that overlapped with the gene in the UCSC Genes track, we would get all of the visible peaks in our output. Additionally, items are output regardless of the amount of overlap. This means that if there is even a single base of overlap between two items, they are included in the output. Configuring output By default, the Data Integrator outputs all fields of all data sources except for the bin column of database tables (a numerical index added by UCSC to speed up database queries). If you are interested in the data from a limited number of fields, you can customize the set of fields to include in the output. After selecting and configuring your data sources as described in the Selecting Data Sources section, click Choose fields... to begin configuring your output. After clicking the button, a pop-up with all the fields from your selected tables will appear: [hgIntegrator choose fields] Uncheck the box next to the field name to exclude it from the output; check the box if you change your mind and want to include it. You can quickly select or deselect all of the fields by clicking Set all or Clear all below the track name. In the UCSC Genome Browser database, detailed information on the annotations for many tracks is stored in extra tables. This information can include things such as identifiers in other databases, transcript status, or other descriptive information. You can select fields from up to four of these related tables for each of your selected tracks. After clicking Choose fields..., you should see a section labeled "Related tables" below each track. This drop-down menu allows you to select and add related tables to your output. [hgIntegrator ] Note that not all tracks in the Genome Browser database will have related tables. Also, some tables may be unavailable when region is set to genome due to the data provider' s restrictions on sharing. If no "Related tables" section appears for a particular selected track, then there are no related tables for this track. After you have finished selecting your output fields and related tables, click Done at the bottom of the dialog box to finish configuring your output. If you want to view your results in the web browser window, simply click Get output. Or, if you want to send your output to a file, check the box next to Send output to file. You can compress your output file using gzip by checking the box next to Compress with gzip (.gz). Then click Get output to begin downloading your file. Note that for wiggle tracks, the "Value" field is averaged across the region of overlap. In the future, we intend to add an option to output all of the data values in that region of overlap. If a single item in the primary track has multiple matches in a secondary track, then the Data Integrator outputs multiple rows. Each row repeats the fields of the primary track item, and then adds fields of the nth matching item from each secondary track. If there are multiple secondary tracks, then their items will appear together in a row of output whether or not they overlap each other. For example, note how items overlap (or not) in the three tracks A, B and C in the image below: [hgIntegrator hgTracks example 2] Two items from track B (B1, B2) overlap the primary item (A). Three items from track C (C1, C2, C3) overlap the primary item (A). The Data Integrator prints 3 rows of output. B1 and C1, although they do not overlap, appear together in the first row of output. B2 and C2 appear together in the second row of output. The third row of output has empty fields for track B followed by C3's fields. This is the output (when A, B and C are 4-field BED custom tracks; colors added for emphasis): #A.chrom A.chromStart A.chromEnd A.name B.chrom B.chromStart B.chromEnd B.name C.chrom C.chromStart C.chromEnd C.name chr1 1200 1800 A chr1 1250 1350 B1 chr1 1425 1525 C1 chr1 1200 1800 A chr1 1550 1650 B2 chr1 1675 1775 C2 chr1 1200 1800 A chr1 1700 1800 C3 
/goldenPath/help/hgSessionHelp.html:Genome_Browser_Session_Help	Sessions User's Guide Contents Introduction Some simple examples Creating a session Creating a session — video demonstration Session details Sharing a session Editing an existing session Displaying your own tracks in a session Deleting a session Lifespan of a session Session gallery Help for Nucleic Acids Research submitters ------------------------------------------------------------------------ Questions and feedback on this User's Guide are welcome. User questions and answers on Sessions and other topics are available in the Genome Browser mailing list. Introduction The Session tool allows you to configure your browser with specific track combinations, including custom tracks, and save the configuration options. Multiple sessions may be saved for future reference, for comparing different data sets, or for sharing with your colleagues. Saved sessions will not be expired, however we still recommend that you keep local back-ups of your session contents and any associated custom tracks. BLAT result tracks persist for at least 48 hours after the last time they are viewed. The creation date of a session can be viewed in the Session Management menu. This date only reflects the initial creation of the Session and is not updated when sessions are edited. Descriptive text can also be added to a session in the Session Details menu. This feature may be accessed via the Session link in the top blue navigation bar in any assembly. To ensure privacy and security, you must create an account and log in before using the session manager. Individual sessions may be designated as either shared or non-shared to protect the privacy of confidential data. To avoid having a new shared session from someone else override your existing Genome Browser settings, you are encouraged to open a new web-browser instance or to save existing settings in a session before loading a new shared session. Note that not all of the Genome Browser mirror sites have all of the session features enabled. This User's Guide provides a few examples that introduce the features of the Session tool, followed by detailed directions on creating, saving, modifying and sharing sessions. You may also wish to reivew two blog posts, How to share your UCSC screenthoughts and Sharing Data with Sessions and URLs for more discussions about sessions. Some simple examples This section contains some example sessions that demonstrate the use of the Session tool. To enable you to view these sessions, we have created a user account with the name Example. Example 1 This example shows the primate (chimp and rhesus) nets for chromosome 2 in the hg17 human assembly — the primate chromosome that fused in humans. We first configured our browser view with the desired settings, and then saved the session so that we could share it. We named our session hg17_chr2_primate. There are several ways for you to view this session: - Manually load and open the session. Open the Session tool. In the Session Management section under the Load Settings heading, enter this information: user: Example session name: hg17_chr2_primate Click the submit button next to the session name box to load the session. To view the session in the Genome Browser, click the Genome Browser link in the top blue navigation bar. - Open a session link sent by email. After we created and saved this session, we could have clicked the Email link to automatically send a message to one or more recipients with the following contents and clickable link: Here is a UCSC browser session I'd like to share with you: http://genome.ucsc.edu/cgi-bin/hgTracks?hgS_doOtherUser=submit&hgS_otherUserName=Examples&hgS_otherUserSessionName=hg17_chr2_primate. By clicking this link, you can open the session in your browser. - Open a session from a local file. Alternatively, if we had saved the browser settings to a local file, we could have simply provided the location of that file for you to load into your browser to view our session. Click here to see such a settings file. This method works best when the file is in a location that you can access from your own computer or network. For this example, you can copy this file and paste it into a file on your own machine, then load it into the Session tool. - Open a session from a URL. Because you do not have access to our file system where this session file resides, it will be easier for you to load it using a URL. To do this, open the Session tool. In the Session Management section under the Load Settings header, enter the URL where this file is located: http://genome.ucsc.edu/goldenPath/help/examples/session_example1.txt Then, click the submit button to load the session settings. To view the session in the Genome Browser, press the Browser link in the Updated Session section. Example 2 This example shows the Human Accelerated Region (HAR1) in the hg18 assembly. Eighteen differences exist in a region of 118 bases between human and all other mammals extending back to the chicken. The two sessions in this example show the same browser position at two levels of detail: Example 2a is zoomed out; Example 2b is zoomed in. To view these sessions in your browser, you can use any of the methods described in Example 1: - Manually load and open the session. Example 2a: user: Example session name: hg18_HAR1 Example 2b: user: Example session name: hg18_HAR1_zoom - Open a session link sent by email. Example 2a: http://genome.ucsc.edu/cgi-bin/hgTracks?hgS_doOtherUser=submit& hgS_otherUserName=Example&hgS_otherUserSessionName=hg18_HAR1 Example 2b: http://genome.ucsc.edu/cgi-bin/hgTracks?hgS_doOtherUser=submit& hgS_otherUserName=Example&hgS_otherUserSessionName=hg18_HAR1_zoom. - Open a session from a local file. Example 2a: Copy the contents of this file to a file on your own machine, then load it into the Session tool. Example 2b: Copy the contents of this file to a file on your own machine, then load it into the Session tool. - Open a session from a URL. Example 2a: Paste this URL into the Session tool: http://genome.ucsc.edu/goldenPath/help/examples/session_example2a.txt Example 2b: Paste this URL into the Session tool: http://genome.ucsc.edu/goldenPath/help/examples/session_example2b.txt Creating a session It is easy to create a session to save or share. Simply configure the Genome Browser as you wish, then navigate to the Session tool by clicking on the My Data pulldown in the top blue navigation bar. Follow these steps to save your session: - Log in to the Genome Browser. To ensure privacy and security, you must create an account and/or log in to use the Session tool. You will not have to repeat the login step unless you sign off from the Session tool or close your Genome Browser. - Create a named session. Scroll down to the Save Settings section of the page. Type a name into the Save current settings as named session box. Choose whether or not you would like to share your sessions with others. If the allow this session to be loaded by others box is checked, anyone will be able to view your Genome Browser settings (including your custom tracks) if you provide them with your user and session name. Note that your session is not automatically available to the general public if you choose this option: you must provide the user and session name to other individuals for them to view it. This helps to ensure the confidentiality of your private data. After naming the session and choosing your sharing option, click the submit button. Your session will then be listed by name under My Sessions. - Save session settings to a file. Alternatively, you can create a file from your session settings that can be saved to your local machine or posted to a URL for access or sharing. To do this, go to the Save Settings section. Type a name into the Save current settings to a local file box. Click the submit button to save or display the file. The session will be saved in plain text (ascii) format by default. To select a compressed format, select one of the options from the file type returned menu before clicking submit . If you simply wish to preview the contents of the file in your browser window, leave the file name blank and click submit . How to back up text-based Custom Track data to a file - Save Custom Tracks. Save a backup of your current browser sessions's custom tracks to your local machine. This backup is intended to be used to restore uploaded text-based custom tracks that would otherwise be lost in case of an unexpected system failure at UCSC. Saving your data is easy and ensures that your hard work is safe and recoverable. To download your custom track data, navigate to the Save Settings section. Click the submit button to the right of "back up custom tracks archive .tar.gz". For each genome assembly, the custom track names will be shown along with individual and total file size. To proceed, click the create custom track backup archive button. All of the custom track data for the active session will be archived and compressed. Large custom tracks may take several minutes to finish. To download the archive to your computer, type in a name for the downloaded archive file and click the download backup archive button. The file will have a ".tar.gz" extension. Note that Safari browsers will unzip the archive leaving you with a .tar file in your Downloads directory. Click the return button to return to the Session Management page. To save viewing settings like track visibilities, highlighting, and sequence positions, use the "Save session settings to a local file" feature mentioned above. - Restore a Custom Track. The downloaded archive file cannot be directly reloaded into the system because it is compressed. To reload custom tracks, untar the downloaded .tar.gz file. The following example assumes you named your archive file 'someproject'. From the command-line, change into your Downloads directory, make a directory for your session, and uncompress your custom track archive. cd Downloads mkdir someproject cd someproject tar -xzf ../someproject.tar.gz If you use the Safari browser, your file will have been uncompressed automatically, leaving just .tar. If this is the case, use this command instead: tar -xf ../someproject.tar The assembly directory contains .ct files, one for each custom track. To restore a single custom track, go to the Custom Tracks tool, select the correct genome database, and click the add custom track button. If you already have existing custom tracks, click the add custom track button, then click the Choose File button next to the top window near the "Paste URLs or data" text. A file selection box will open. Navigate to the directory where the desired custom track .ct file is, choose it, and click the Submit button. If an optional help text or HTML page was saved with the custom track, you will see a corresponding custom track .html file. You can restore this by selecting the Choose File button near the "Optional track documentation" text. A file selection box will open. Navigate to the directory where the .html file is located, choose it, and click the Submit button. - Restore Multiple Custom Tracks. Restore all the custom tracks in an assembly directory by concatenating the .ct files together. From the command-line, change directory to your genome assembly and concatenate all of the .ct files into on hg38_all.ct file: cd hg38 cat *.ct > hg38_all.ct Compress the concatenated file for faster upload, creating hg38_all.ct.gz with the following command: gzip hg38_all.ct Navigate to the Custom Tracks tool and select the big concatenated compressed file (hg38_all.ct.gz in our example). Note that large custom track sets may require a long time to upload. Once uploaded, click the Go button on the next page to view your custom tracks in the Browser. You will still be able to save and share your Sessions online, which will include your Browser Settings and Custom Tracks. This feature is solely for backing up Custom Tracks as files. Creating a session — video demonstration []       Visit our Video Page. Visit our YouTube channel for more videos. Opening a saved session When you save a session, it is added to the My Sessions list on the Session page. Each session entry is listed by name and offers the following options to open, share, and manipulate it: - Session name. Click the session name to view it in the Genome Browser. - View/edit details. Click the details button to edit the session description and view session details such as creation date/time, assemblies with custom tracks and more. - Delete the session. Click the delete button to permanently remove this session from the list. - Share with others: Check this box to allow others to access this session. By default, this option is unchecked, which limits access to only the session owner. - Post in public listing: Check this box to add your session to the list of Public Sessions. Sessions in the listing will be available to be loaded and viewed by the world. - Email. Click this link to email this session to a colleague. Session details Each session has an associated details page that you can click into from the Session Management menu. The Session Details menu allows you to edit the Session Name, to add descriptive text and to change whether or not the session is shared with others. Like the Session Management menu, if you click "use" that session will be loaded as the current session and if you click "delete" the session will be deleted. The "Created on" date reflects the date that the session was originally created and will not be updated to reflect any edits. Sharing a session Shared vs. non-shared data When you create a session using the Session tool, you may designate it as either shared or non-shared. By default, new sessions are created as shared and must be explicitly changed to non-shared status. Shared sessions can be opened by other Genome Browser users to whom you've provided one of the following: - the user name and session name of the saved session - access privileges to a local file that contains the saved session information - the URL of a web-accessible session settings file Sessions which you've added to the list of Public Sessions will available to the world. Note that unless you've added them to this list of Public Sessions, your shared sessions will not be available in a general way to other Genome Browser users; they will need at least one of these access methods. If you choose to keep your session private, other users of the Genome Browser will not be able to access your data or browser configuration. Any confidential data or locations of interest that you are working with will be safe from viewing by others. The most secure way to control your session is to save the settings to a local file, then deny access to that file by others. Sharing your session with others There are five ways to let others know about your saved sessions: - Save the session URL. Immediately upon saving a new session, the top of the page offers a Browser hyperlink. Additionally, each session entry in the My Sessions list has a Browser hyperlink. Click either Browser link to open the Genome Browser with the session loaded. You can obtain the URL of the Genome Browser page by capturing the Browser hyperlink via right-click before you proceed to the Browser graphical view. You can then store the URL, create a bookmark or share the link with others. The UCSC site and supported mirrors display short session links similar to the following: http://genome.ucsc.edu/s/YourUserName/YourSessionName Previously, longer links were used: http://genome.ucsc.edu/cgi-bin/hgTracks?hgS_doOtherUser=submit&hgS_otherUserName=YourUserName&hgS_otherUserSessionName=YourSessionName System administrators of mirrors can introduce Apache redirects from the shorter links that will redirect to the longer links and enable the links to display for new sessions by setting hgSession.shortLink=on in the mirror's configuration files. Both links can be modified to include URL parameters, such as position=chr10:69,644,222-69,644,999 in these examples: http://genome.ucsc.edu/cgi-bin/hgTracks?hgS_doOtherUser=submit&hgS_otherUserName=view&hgS_otherUserSessionName=clinicalzoom&position=chr10:69,644,222-69,644,999 http://genome.ucsc.edu/s/view/clinicalzoom?position=chr10:69,644,222-69,644,999 - Email a session link. Each session entry in the My Sessions list also has an Email link. Click this link to automatically invoke your email tool with a message containing the Genome Browser URL, which you can then send to others. - Share a session settings file. If you have saved your settings to a local file, you can give others access to the file, or email the file to them as an attachment and instruct them to load it using the Session tool. - Share a web URL. If you have saved your settings to a file on a web server, you can provide a link like this to others: http://genome.ucsc.edu/cgi-bin/hgSession?hgS_doLoadUrl=submit&hgS_loadUrlName=MyUrl where MyUrl is the URL of your settings file, e.g., http://www.mysite.edu/~me/mySession.txt. In this type of link, you may replace "hgSession" with "hgTracks" to proceed directly to the Genome Browser. - List it on the Public Sessions page. The "Public Sessions" tool allows you to post and share your exciting and interesting Browser snapshots with the world. After having saved your session, you can add it to this public listing by checking the box in the column under "post in public listing?". People can then find your sesssion by entering a search term. You can even create a gallery of Public Sessions related to your search by using a unique string in your descriptions. Opening a shared session If you open a shared session while viewing the Genome Browser, it is possible to lose all of your own browser settings. That is, the settings for the newly-opened session will take precedence over your existing settings and will replace them. If you wish to preserve your original settings, you should first save your own settings as a session before opening a new session, or open a new tab or window in your internet browser before loading the new session. There are four ways to open a shared session, depending on what information you have about the session. The instructions below assume that you want to replace your current session the new session. Be sure to preserve your original session first if you don't want to overwrite it. - Open a session from an email link. If you receive an email message with a link to a colleague's shared session, simply click on the link to view the Genome Browser with the session settings. - Open another user's session. If you know the name of another user's shared session you can type in the user and session name in the "Restore Settings" section and click "submit" This will generate an "Updated Session" message and you can click on the Browser link to load the browser with the settings saved in this session. - Open a session from a settings file. Open the Session tool, then scroll down to Restore Settings in the Session Management section. Click Choose File to find the file on your computer. Click submit to display the Genome Browser using these session settings. - Open a session specified by a URL. Open the Session tool, then scroll down to Restore Settings in the Session Management section. Type in the URL in the Use settings from a URL box, then click submit to display the Genome Browser using the new session settings. Editing an existing session It's easy to make changes to an existing session. Reconfigure the Genome Browser as you wish, then resave the session with the same name. The Session tool will warn you that you are about to overwrite an existing session. You can also edit any descriptive text associated with your session as well as whether or not the session can be shared in the Session Details menu. Note that editing a session will not alter the creation date listed in the Session Management menu. If you previously shared this session with others, they will not see the changes until they reload your newly-edited session. Displaying your own tracks in a session In addition to displaying standard UCSC tracks in your session, you can also display the following user-generated tracks: - Custom Tracks - Genome Graph tracks Before you create and save your session, be sure to upload your Custom Track or Genome Graph track. These user-generated tracks associated with a saved session will not expire. BLAT results always have a lifespan of 48 hours, even if they are part of a session. However, if you generate a custom track from your BLAT results, they will be saved in your session. Deleting a session In the Session Management section under My Sessions, press the delete button next to the session name you would like to delete. This will permanently delete all details of the session from the UCSC server. Any saved links to that session will no longer work. No other user can delete your saved sessions, even if you have provided access to your sessions to them. Other users simply have a copy of your session. Unlike most other browser preferences, the session settings are not saved in your Genome Browser "cart". Therefore, if you choose to reset the Genome Browser, it will not delete your saved sessions. Lifespan of a session Your saved sessions will not be expired and will available you (and others if you share them) until you delete them. We have discontinued our previous policy of removing saved sessions and associated custom track data after four months. However, note that the UCSC Genome Browser is not a data storage service; please keep a local backup of your session contents and custom track data. Session gallery The Session Gallery is a collection of track views that help highlight viewing different topics in the browser. The sessions in the Session Gallery were created in the browser and then saved to a local file, which was then uploaded to an online location. This allows creating a single link, such as http://genome.ucsc.edu/cgi-bin/hgTracks?hgS_doLoadUrl=submit&hgS_loadUrlName=U, where U is the URL of the session file, e.g., http://www.mysite.edu/~me/mySession.txt, enabling users to maintain external control of the content file for easy update. Help for Nucleic Acids Research submitters The Nucleic Acids Research (NAR) journal now requires manuscript submissions to contain a private Session link to your data in the UCSC Genome Browser that allows reviewers to access data. These instructions will show you how to upload, view, and share your data. Viewing your data on the Genome Browser You can view your own private data by uploading your annotation files to the Genome Browser as custom tracks; visit our custom track help page to learn more. To summarize the steps to upload your data, you will need to: 1. Ensure the data file is formatted correctly. 2. Create a track line for your custom track. 3. Load the custom track by adding your track line to our Custom Tracks page. 4. View the data in the Genome Browser. Custom track examples Creating the track line may be the most challenging step since many configuration options exist. The track line begins with the track keyword, followed by one or more attribute=value pairs where the order of the attributes does not matter. Here are some examples: BAM custom track The simplest example of a BAM custom track is the following track line: track name="My BAM" type=bam bigDataUrl=http://www.mysite.edu/~me/my_sorted.bam In the example above, the name attribute defines the name of your custom track. The second attribute, type, is required for some data types, but not limited to: BAM, bigBed, WIG, bigWig, and VCF data types. The last attribute, bigDataUrl, is required for remotely hosted data types such as BAM, CRAM, bigBed, bigWig, and VCF. Adding more attribute=value pairs can further customize the display. Here is a custom track that uses the visibility and description atrributes: track type=bam visibility=dense name="My BAM" description="Example from the ENCODE RNA-seq CSHL track" bigDataUrl=http://hgdownload.soe.ucsc.edu/goldenPath/hg19/encodeDCC/wgEncodeCshlLongRnaSeq/wgEncodeCshlLongRnaSeqA549CellPapAlnRep1.bam There are also options to configure the display of your BAM files, such as a density plot feature that will dynamically process the underlying BAM into a wiggle signal. ------------------------------------------------------------------------ bigWig custom track A bigWig file is useful when trying to display dense, continuous data. Read more on the bigWig track format help page. Here is an example bigWig track that is colored red, instead of the default black color, that can be pasted directly into the Custom Tracks Page: track color=255,0,0 name="HeLa-S3 nucleus minus signal" description="RNA Subcellular CAGE Localization from ENCODE/RIKEN" type=bigWig visibility=full bigDataUrl=http://hgdownload.soe.ucsc.edu/goldenPath/hg19/encodeDCC/wgEncodeRikenCage/wgEncodeRikenCageHelas3NucleusPapMinusSignalRep1.bigWig There are also options to configure the display of your wiggle tracks, such as changing the track height or type of graph. Creating a session for NAR publications After creating your custom tracks and viewing your data on the Genome Browser, you can save all of your tracks and settings to a snapshot of the Genome Browser called a session. You can easily save a session by following these five steps: 1. Configure the Genome Browser to your preference Make sure the display of your custom tracks is to your liking on the Genome Browser. 2. Navigate to the sessions page Once you are satisfied with the display, go to the My Sessions page by either: - Going to My Data -> My Sessions from the navigation bar. - Using the "s then s" keyboard shortcut when viewing the main page of the Genome Browser. 3. Login to the UCSC Genome Browser You must sign in to be able to save named sessions which will then be displayed with Browser and Email links. 4. Save your session Go to the Save Settings section and in the Save current settings as named session text box, and enter a name for your session. When saving the session, be sure to have the "allow this session to be loaded by others" option checked and then click submit. - You should then be able to copy and share a link similar to the following: http://genome.ucsc.edu/s/YourUserName/YourSessionName 5. Edit the session description Once the session is created, under the "view/edit details" column you can click the view/edit button to add a description to the session. 6. Publish your session as a Public Session for public discovery If you eventually make a Public Session of your session after your paper is published, and provide a detailed description, anyone can find it by searching for terms you share. - For example, navigate to the Public Sessions page and search "NAR" to see some example sessions. Or click this link, which adds ?search=NAR to the URL. - As a Public Session, your session can then be found by anyone entering a search term, where you can even create a gallery of Public Sessions related to your search by using a unique string in your descriptions. - Here is an example "gallery" of all sessions that have the word "sessionView" in their descriptions: http://genome.ucsc.edu/cgi-bin/hgPublicSessions?search=sessionView Sharing your session link After saving your session, you will now be able to share your session link with others. There are three different ways that you can share your session: 1. Immediately after creating a new session, the top of the page offers a Browser hyperlink that will allow you to view the newly saved session. You can right-click the Browser link to save the URL. 2. Alternatively, you can right-click the session name and then click Copy Link Address to save the session URL. 3. Adding your session to the Public Sessions page will provide a useful way of sharing your session with the world in the long run. You can add your session to our Public Sessions by selecting the "post in public listing?" checkbox. Editing the session URL You can edit URLs to directly go to different parts of the Genome Browser such as by changing hgSession to hgTracks, e.g., http://genome.ucsc.edu/cgi-bin/hgTracks?hgS_doOtherUser=submit&hgS_otherUserName=brianlee&hgS_otherUserSessionName=hg19.rela Having hgTracks in the session URL will take the viewer to the main page of the Genome Browser instead of the sessions page. Using track hubs to organize custom tracks Creating a track hub allows you to group and organize your annotation tracks, but is limited to compressed binary indexed formats that can be remotely hosted. This will require the use of a web-accessible location to load the track hub in the UCSC Genome Browser. If your institution does not provide web space for you, you can look into hosting your binary files at a couple different web hosting services, such as CyVerse or figshare. If you would like to learn more about using track hubs, please read our Track Hub help page. Genome Browser inquiries If you have any questions regarding the creation of your custom track or session, please feel free to contact us. Before submitting a question, we strongly encourage you to search our mailing list archives, our website, and our wiki for the answer. 
/goldenPath/help/hgNearHelp.html:Gene_Sorter	UCSC Gene Sorter User's Guide Contents Introduction Getting started Understanding the Gene Sorter display Configuring the Gene Sorter display Filtering the gene display Displaying sequence and text-based output ------------------------------------------------------------------------ Search the Genome Browser help pages:   Questions and feedback are welcome. Introduction Genes function and evolve together.To understand a gene, you often need to understand an entire gene family. Many such families are already known and described, such as the HOX family that mediates many aspects of limb and brain development and the Cytochrome P450 family that is central to the metabolism of many medications. One easy way to identify well-known relatives of a gene is by looking for genes with name similarity, because biologists tend to use similar names for similar genes. However, scientists only partly understand the function of perhaps one-third of the genes of the genome; therefore, other techniques for grouping genes into families are necessary. The UCSC Gene Sorter is an excellent resource for exploring gene families and the relationships among genes. This tool displays a table of genes within a selected genome that are related to one another. Several different relationships may be explored: protein-level homology, similarity of gene expression profiles, or genomic proximity. The Gene Sorter supports searches on a variety of terms and phrases, including the gene name, the UniProtKB protein name, a GenBank accession, or a word or phrase present in a gene's description. The gene family display is highly configurable, allowing the user to control the order and number of columns, the number of rows, and the genes displayed. The tool provides several output formats, including a simple tab-delimited format that may be imported into a spreadsheet or a relational database. An important use of the Gene Sorter is to gather together a collection of genes that share similar properties for statistical analysis. For instance, one might want to examine promoter regions of genes that share a similar expression pattern or look for protein sequence motifs in genes that share similar GO annotations. One of the most powerful features of the Gene Sorter is its filtering capabilities. The filter enables the user to quickly select an interesting subset of the 25,000 genes in the genome based on a variety of detailed and flexible selection criteria. For example, the filter may be used to select all human genes over-expressed in the cerebellum that have GO-annotated G-protein coupled receptor activity. The Gene Sorter was designed and implemented by Jim Kent, Fan Hsu, David Haussler, and the UCSC Genome Bioinformatics Group. This work is supported by a grant from the National Human Genome Research Institute and by the Howard Hughes Medical Institute. Getting started To begin using the Gene Sorter, you will first have to select a genomic region and the type of gene relationship you wish to display. You may also want to change some of the Gene Sorter's configuration settings to tailor the display to your research needs. These configuration options are described in Configuring the Gene Sorter display. Starting the Gene Sorter 1. Open the Gene Sorter home page. 2. Specify the genome and assembly you wish to view by selecting the appropriate options from the genome and assembly pull-down menus. 3. Type a term or phrase into the search text box to determine which genes will be displayed in the browser. Valid search terms (with human genome examples) include: - a gene name (HOXA9) - a UniProtKB protein name (HXA9) - a word or phrase that occurs in the description of a gene (MAP kinase) - a GenBank mRNA accession (U14680) 4. Choose the gene relationship that you would like to examine by selecting an option from the sort by pull-down menu. Genes will be sorted in order of proximity to the chosen gene, based on one of the following criteria: - Expression (GNF Atlas1) -- similarity in gene expression, based on GNF Atlas 1 data - Protein Homology - BLASTP -- similarity in protein homology, based on the BLASTP E-value - Protein Homology - Rankprop -- similarity in protein homology, based on the Rankprop algorithm - Protein Homology - PSI-BLAST -- similarity in protein homology, based on the PSI-BLAST E-value - Pfam Similarity -- similarity based on number of shared domains - Gene Distance -- absolute distance (left or right) on the chromosome from the selected gene - Chromosome -- list sorted by chromosomal location - Name Similarity -- similarity to the name of selected gene, based on the first several characters of the name - Alphabetical -- list sorted by gene name - GO Similarity -- number of Gene Ontology (GO) terms shared with selected gene 5. Choose the number of items to display from the display pull-down menu (the default is 50). 6. Press the Go! button to display your search results. Understanding the Gene Sorter display The main page of the Gene Sorter displays a table containing rows of genes and associated attributes. In most cases, the currently-selected gene is shown at the top of the list, highlighted in light green. The remaining genes are ordered relative to the selected gene based on the sort criteria specified in the sort by menu. For example, in a table sorted by gene distance, the genes are listed in order of greater to lesser chromosomal proximity to the selected gene. The initial Gene Sorter display shows only a default subset of the columns available. The set of columns may be expanded, reduced, and rearranged by using the Gene Sorter's configuration utility. To view information about the data shown in the column, click on the column's label. To select a different gene in the table, click on the name of the gene. The Gene Sorter will move the gene entry to the top of the list and highlight it. The remaining genes will be reordered relative to the new selection. Column descriptions (listed in alphabetical order) - # column: Displays the position of each gene in the table, which is useful when examining data in tables with many rows. Clicking on the gene number selects it and moves it to the top of the list. - 3' UTR Fold column: Shows the estimated energy in kcal/mol of folding the 3' UTR of a gene into the best predicted secondary structure. The energy calculations and secondary structure predictions were obtained from the RNAfold program, which is part of the Vienna RNA Package. - 5' UTR Fold column: Shows the estimated energy in kcal/mol of folding the 5' UTR of a gene into the best predicted secondary structure. The energy calculations and secondary structure predictions were obtained from the RNAfold program, which is part of the Vienna RNA Package. - Abundance column (Yeast): Displays protein abundance information, as reported in Ghaemmaghami et al. Global analysis of protein expression in yeast, Nature 425(6959), 737-741 (2003). For more information, see the Yeast GFP Fusion Localization Database. - Arbeitman et al. 2002 Life-Cycle Expression Data column (Fruitfly): Shows the median ratio of gene expression in various phases of the fly life cycle relative to the expression of the gene in mixed egg-to-adult cultures. The level of detail may be increased or decreased with the configuration utility. See the Expression columns description for more information about the display and configuration of this column. For more details about the experiments and methods used to create this data, click on the column's label. - BLASTP Bits column: Shows the bit score measure of protein similarity between a gene and the selected gene. The greater the similarity between two proteins, the higher the bit score. For more information about how the bit score was calculated, click on the Bits column label. - BLASTP E-Value column: Shows the Blastp E-value (expectation value) between each gene and the selected gene. The greater the similarity of two proteins, the lower the E-value is. Identical long proteins have an E-value of zero. Formally, an E-value is the number of other known genes that are expected to have at least this level of homology by chance. An E-value of less than 0.1 can be safely interpreted as the probability that a match this good would occur merely by chance. For more information about how the E-value was calculated, click on the E-Value column label. Clicking on a gene's E-value displays an alignment between the gene and the selected (highlighted) gene. - C. elegans column: Shows the best Blastp match to the WormBase protein set. Clicking on the ID number displays the corresponding WormBase database record. - Coding SNPs column: Shows the Simple Nucleotide Polymorphisms (SNPs) located in the coding region of each gene. Clicking on a SNP ID displays the SNP record associated with the gene in NCBI's dbSNP. - Description column: Shows a brief description of each gene taken from its mRNA record. Clicking on a description displays a page showing additional information and links for the gene. - Drosophila column: Shows the FlyBase ID of the best Blastp match to the FlyBase protein set. Clicking on the FlyBase ID displays the corresponding FlyBase report. - Ensembl column: Shows the Ensembl transcript ID associated with the gene. Ensembl is an automatic gene prediction pipeline and a major genome database and web site run by the Sanger Institute and the European Bioinformatics Institute (EMBL-EBI). It is especially effective at mapping genes that are known in one organism to another organism. Compared to other gene predictions, those of Ensembl tend to have high specificity but low sensitivity to genes not already associated with characterized mRNA or protein sequence. Clicking on an Ensembl ID displays the Ensembl GeneView page for the gene. - Entrez Gene (Human): Formerly LocusLink. Shows the NCBI Entrez Gene ID associated with the gene. Clicking on the entry displays the Entrez Gene record. If the record shows a link to the Online Mendelian Inheritance in Man (OMIM) -- indicated by an orange square with an "O" inside it -- an OMIM record is available for the gene. - Exon Count column: Displays the number of exons in the gene (coding and non-coding). - Exp Delta column: Shows the similarity of the expression of each gene to the selected gene. Genes with identical expression profiles have a value of 0. This column shows data only for the 1000 genes (including splicing variants) that have the most similar expression profiles. For more information about how expression distance was calculated, click on the Exp Delta column label. - Expression columns: Show the ratio of expression of the gene in selected tissues or life cycle stages to the expression of the gene overall. A gene that is more highly expressed is colored red, and a less expressed gene is shown in green. The values are colored on a logarithmic scale. This coloring is standard, but is the opposite of what an inexperienced user might expect: in this case, red means go and green means stop! Black indicates that a gene is neither over nor under expressed in the tissue. Uncolored boxes (white on most browsers) represent missing data. Various attributes of the expression column displays may be configured using the configuration utility. Depending on the organism, the user may adjust the color scheme and brightness, toggle between expression ratio and absolute expression values, and increase or decrease the level of detail shown. In particular, color-blind users may wish to switch the coloring from red/green to yellow/blue. For more information about the selection criteria used for expression columns, click on the column's label. - GenBank column: Shows the GenBank RefSeq or mRNA accession number associated with the gene. Clicking on the accession number displays the GenBank record associated with it. - Gene Ontology column: Shows the Gene Ontology (GO) terms associated with the gene. GO terms are words from a controlled vocabulary assigned to a gene by human curators. Clicking on a GO term displays the associated Gene Ontology Consortium database record. - Genome Position column: Shows the chromosomal location of each gene in the genome. Clicking on a chromosomal position displays the gene at that location in the UCSC Genome Browser. - GNF Atlas2 column (Human): Shows the ID of the probe in the GNF Atlas 2 that overlaps most with the selected gene. The GNF Altas 2 is based on two Affymetrix chips: the U133A and a custom-designed GNF1H chip. - GNF Delta column: Shows the similarity of the expression of each gene to the selected gene. Genes with identical expression profiles have a value of 0. This column shows data only for the 1000 genes (including splicing variants) that have the most similar expression profiles. - GNF U74a, GNF U74b, GNF U74c columns (Mouse): Shows data from the Mouse Gene Expression Atlas from the Genomics Institute of the Novartis Research Foundation (GNF) on the Affymetrix U74a, U74b, and U74c chips. By default, the columns display the median ratio of expression in a specific set of tissues relative to the expression of the gene overall. The level of detail may be increased or decreased with the configuration utility. Currently, the full spectrum of tissues is available only on the U74a chip. See the Expression columns description for more information about the display and configuration of this column. For more details about the experiments and methods used to create this data, click on the column's label. - GNF U95 column (Human): Shows data from from the GNF Expression Atlas on the Affymetrix U95 chip. By default, the column displays the median ratio of expression in a specific set of tissues relative to the expression of the gene overall. The level of detail may be increased or decreased with the configuration utility. See the Expression columns description for more information about the display and configuration of this column. For more details about the experiments and methods used to create this data, click on the GNF U95 column's label. - Human column (Mouse): Shows that best Blastp match to the known genes protein set from the UCSC Human Genome Browser database. Clicking the accession number displays the gene in the UCSC Human Genome Browser. - Kim-Lab Life-Cycle Expression Data (All) column (C. elegans): Shows the ratio of gene in all phases of the worm life cycle relative to the expression of the gene in mixed wild-type adult cultures. See the Expression columns description for more information about the display and configuration of this column. For more details about the experiments and methods used to create this data, click on the column's label. - Kim-Lab Life-Cycle Expression Data (Median) column (C. elegans): Shows the median ratio of gene expression in selected phases of the worm life cycle. See the Expression columns description for more information about the display and configuration of this column. For more details about the experiments and methods used to create this data, click on the column's label. - Max GNF Atlas 2 column: Shows the maximum absolute expression level for any tissue in the GNF Gene Expression Atlas 2. Most of the values fall in the zero to 50,000 range, but a few outliers may range as high as 200,000. A value of less than 20 indicates the expression could not be detected in any tissue at levels significantly above the cross-hybridization controls. - Max GNF U95 column (Human): Shows the maximum absolute expression level for any tissue. Most values range between zero and 30,000, but a few outliers may range as high as 52,000. A value of less than 20 indicates the expression could no be detected in any tissue at levels significantly above the cross-hybridization controls. - Max Rinn Sex column (Mouse): Shows the maximum expression value in adult male and female mouse tissue as described in (Rinn et al., Developmental Cell, 2004). For more information about the methods used to generate these data, click on the Max Rinn Sex column header. - Module column (Yeast): Shows the predicted regulatory module (a combination of transcription factor binding sites) that regulates the gene. The approach underlying this annotation is described in Segal, E. et al., Genome-wide discovery of transcriptional modules from DNA sequence and gene expression, Bioinformatics 19(Suppl 1), i273-i282 (2003). For more information about the methods used, click the Module column header. To view genes that share a regulatory module, select the Regulatory Module option from the sort by menu. - MOE430 ID column (Mouse): Shows the Affymetrix ID from the MOE430 series of chips (A & B) that best corresponds to each gene. For more details about the experiments and methods used to create this data, click on the column's label. - Mouse column (Human): Shows the accession number of the best Blastp match to the known genes protein set in the UCSC Mouse Genome Browser database. Clicking on the accession number displays the gene in the UCSC Mouse Genome Browser. - Name column: Displays the name of the gene. When possible, the HUGO Gene Nomenclature Committee (HGNC) name is shown. If the gene does not yet have an HGNC name, the GenBank accession number of the associated RefSeq or mRNA record is shown instead. Clicking on the gene name selects it and moves it to the top of the list. - % ID column: Shows the percentage identity at the protein level between the gene and the selected gene. For more information about how the % ID was calculated, click on the % ID column label. - PDB column: Displays all Protein Data Bank (PDB) IDs associated with the gene. PDB is a database of proteins with known 3-D structures. In some cases these records will correspond to only a fragment of the gene. In other cases the PDB record may include other molecules with which the protein interacts. Clicking on a PDB entry displays the associated PDB Structure Explorer page. - Pfam Domains column: Shows a list of Protein Family (Pfam) domains contained in the gene product. Clicking on a domain displays the associated Pfam record. - PSI-BLAST E-Value column: Shows the PSI-BLAST E-value (expectation value) between the UniProtKB or TrEMBL protein associated with the gene and the protein associated with the selected (highlighted) gene. The greater the similarity of two proteins, the lower the E-value is. Identical long proteins have an E-value of zero. For more information about how the E-value was calculated, click on the E-Value column label. Clicking on a gene's E-value displays an alignment between the gene and the selected (highlighted) gene. - Rankprop column: Displays protein similarity scores assigned by the Rankprop algorithm. The scores reported in this column range from zero to one, with one being the most significant. Currently, Rankprop does not report an E-value statistic. For more information on this algorithm, click on the Rankprop column label. - RefSeq column: Displays the NCBI RefSeq accession associated with the gene, if available. RefSeq genes are a non-redundant set of high-quality mRNA sequences. Clicking on the accession number displays the NCBI Entrez Gene record for a RefSeq accession. If the record shows a link to the Online Mendelian Inheritance in Man (OMIM) -- represented by an orange box with an "O" on it -- an OMIM record is available for the gene. OMIM is often an excellent source of human-curated information about human genes. - Regulatory Motif column (Yeast): Shows the predicted transcription factor binding sites that correlate with expression of the gene. regulatory module (a combination of transcription factor binding sites) that regulates the gene. The approach underlying this annotation is described in Segal, E. et al., Genome-wide discovery of transcriptional modules from DNA sequence and gene expression, Bioinformatics 19(Suppl 1), i273-i282 (2003). For more information about the methods used, click the Module column header. - SGD ORF: Shows the Saccharomyces Genome Database (SGD) ID associated with the gene. - SP Acc column: Shows the UniProtKB protein accession of a gene. Clicking on an entry displays the corresponding UniProtKB NiceProt view of the protein. - Superfamily column: Displays a list of Structural Classification of Protein (SCOP) superfamilies associated with a protein. The gene set was mapped to SCOP superfamilies using the Superfamily HMM library. Clicking on an entry displays the associated Superfamily record. - U133 ID column (Human): Shows the Affymetrix ID from the HG-U133 chip that best corresponds to each gene. For more information about the selection criteria used for this column, click on the U133 ID column's label. - U133Plus2 ID column (Human): Shows the Affymetrix ID from the HG-U133 Plus 2.0 chip that best corresponds to each gene. For more information about the selection criteria used for this column, click on the U133Plus2 ID column's label. - U74 ID column (Mouse): Shows the Affymetrix ID from the U74 series of chips (a, b, c) that best corresponds to each gene. For more information about the selection criteria used for this column, click on the U74 ID column's label. - U95 ID column (Human): Shows the Affymetrix ID from the HG-U95 chip that best corresponds to each gene. For more information about the selection criteria used for this column, click on the U95 ID column's label. - UCLA Long Expression column (Human): Shows UCLA expression data from normal tissues on the U133 chip. This column shows the ratio of expression of a gene in the entire tissue set relative to the expression of the gene overall. See the Expression columns description for more information about the display and configuration of this column. For more information about the methods used to generate this data, click on the UCLA Long Expression column's label. - UCLA Short Expression column (Human): Shows UCLA expression data from normal tissues on the U133 chip. This column shows the ratio of expression of a gene in a particular subset of tissues relative to the expression of the gene overall. See the Expression columns description for more information about the display and configuration of this column. For more information about the methods used to generate this data, click on the UCLA Short Expression column's label. - UniProtKB column: Displays the UniProtKB protein name of each gene, if it is available. Otherwise, it shows the primary accession number. Clicking on a protein name or accession displays the corresponding UniProtKB NiceProt view of the protein. - WormBase column (C. elegans): Shows the ORF name associated with each gene. Clicking on an ORF name displays the associated WormBase record. - Yeast column: Shows the best Blastp match to the Saccharomyces Genome Database (SGD) protein set. Clicking on the yeast ID displays the corresponding SGD record. - Zebrafish column: Shows the Ensembl peptide ID of the best Blastp match to the Ensembl gene predictions on the zebrafish genome. Clicking on the Ensembl peptide ID displays it in Ensembl Zebrafish Protein View. Configuring the Gene Sorter display The Gene Sorter is highly configurable, allowing you to fine-tune the display to show just the genes and data columns in which you're interested in an order that best suits your research needs. Most of the configuration is controlled through settings on the Configuration page, accessed via the configure button at the top of the Gene Sorter page. Changing the number of rows displayed To increase or decrease the number of rows shown in the table, pick a new value from the display pull-down menu, then click the Go! button. Changing the number of columns displayed By default, the Gene Sorter shows only a small subset of the table columns available for the genome. You can view the full list of columns, or add or remove columns from your display, on the Configuration page. The configuration table shows all the columns available for the currently-selected genome, listed in left-to-right display order. To add or remove a column from the Gene Sorter display, click the On checkbox to toggle the setting (a check indicates that the column is displayed). To quickly change the On settings of all columns, click the Hide All or Show All button at the top of the page. Click the Submit button to display the changes in the Gene Sorter. Changing the column positions In addition to adding or removing columns, it is also possible to move the columns to the left or right within the Gene Sorter table. The order of the column names in the configuration table indicates the current relative position of the columns in the Gene Sorter display from left to right. To shift a column one position to the left, click the up arrow in the item's Position column. Similarly, click the down arrow to shift a column to the right. When you have finished making changes, click the Submit button. Changing the expression colors By default, the gene expression ratios are shown using a red/green color scheme, where red indicates a gene that is more highly expressed and green corresponds to less expression. Color-blind users may find it helpful to switch the coloring from red/green to yellow/blue. To do so, select the "yellow high/blue low" option from the Expression ratio colors pull-down menu on the Configuration page, then click the Submit button. Changing the brightness of expression colors To increase or decrease the brightness of the colors in an expression column, edit the brightness value for the corresponding entry in the configuration table. Values greater than 1.0 increase the brightness, while those less than 1.0 dim the color. Click the Submit button to display the new values. Changing the type of tissue data shown in expression columns By default, the expression columns show the median ratio of expression of a gene in a small selected set of tissues. Use the the tissues pull-down menu to configure the tissue display for the column. The "all replicas" option will show the value of each individual experimental replica of each tissue. The "median of replicas" option displays a single value for each tissue that represents the median of all replicas for that tissue. Toggling between ratio and absolute expression values By default, expression columns show the ratio of expression of a gene relative to expression of the gene overall. To view absolute expression values instead, select the "absolute" option from the values pull-down menu. Displaying splicing variants By default, the Gene Sorter shows only one splicing variant: the one that produces the largest protein. To show all splicing variants, click the Show all splicing variants checkbox. Note that in most cases, the column values (and sometimes the names) will be identical across variants. Restoring the default settings At any time during your Gene Sorter session, you can restore the Gene Sorter table to its default layout by clicking the Default button on the Configuration page, then clicking Submit. Saving a configuration for future use The Gene Sorter configuration utility allows you to store multiple configurations for use in future sessions. This feature is particularly useful if you require different layouts for different research uses. To save the current configuration of the Gene Sorter layout, click the Save button on the Configuration page. Type in a name for the configuration in the text box at the top of the page, then click Save. Loading a previously-saved configuration Once you have saved a configuration, you can load it back into your Gene Sorter in a future session. To load a configuration, click the Load button on the Configuration page. The Gene Sorter will display a list of the names of your saved configurations. Click on a name to highlight it, then click Load to reconfigure your Gene Sorter based on the saved settings. Viewing a list of saved configurations To display a list of configurations that you have saved, click the Save button on the Configuration page. If you have any saved configurations, the Gene Sorter will display an Existing Setups list that shows the configuration names. To permanently remove a configuration from the list, click on the name to highlight it, then click the Delete Existing Setup button. Filtering the gene display The Gene Sorter's gene filtering capabilities provide a versatile way to fine-tune the display to show just the genes in which you are interested. Filters are applied to individual gene fields, and may be combined to increase the specificity of the search. To access the Filter page, click the filter button at the top of the Gene Sorter page. At any time during the filter setup process, you can click the List Names button on the Filter page to view a list of genes that will be returned when the current filter settings are applied to the genome. You may find this list helpful in fine-tuning the filter. Filtering based on matching one or more terms Filters based on names, IDs, or other words restrict the display to only those genes that match one or more terms typed into the search text box. Examples of values that can be filtered on this basis include the gene name, RefSeq accession number, gene description, coding SNPs, and GO terms. This search supports wildcard matching on "*" and "?". Multiple terms must be separated by a space or tab. For example, the search criteria "HOXA9 FOX*" on the gene name field returns the gene named HOXA9 and any gene whose name begins with the letters "FOX". When searching on fields that consist of values containing more than one word (GO terms, coding SNPs, Pfam domains, and gene descriptions), the multi-word elements must be enclosed in single quotes. For instance, a search on the description phrase "forkhead box protein" should be entered as "forkhead box protein". Use the "any" and "all" options to determine whether the search should return any gene that matches any term ("any") or only those genes that match all terms ("all"). To facilitate searching on multiple terms, the Gene Sorter provides the option to paste in or upload a list of search terms. To paste in a list of terms, click the filter's Paste List button, then paste or type the terms into the text box. Terms must be separated by a space, a tab, or be entered on separate lines, and may not include wildcards. When you have completed the list, click the Submit button to return to the main Filter page. The file upload utility - accessed via the Upload List button - has a similar functionality. Filtering based on numerical ranges Several of the gene fields can be filtered by specifying a numerical range within which the value must fall. Examples of fields in this category include expression ratios, Blastp data, and genome position. To use this type of filter, enter the minimum and maximum values delimiting the range in which you are interested. In some cases, the range of valid values is indicated in the filter box. The genome position filter requires the name of a chromosome (in the format chrN) in addition to the chromosomal start and end positions. To list all genes on a chromosome, enter only the chromosome name. Expression filters include "any" and "all" options to determine whether the search should return a gene if any of the tissue expression values meet the minimum and maximum criteria ("any") or only if all tissue expression values meet the search criteria ("all"). Saving filter settings The Gene Sorter provides a mechanism for saving filter settings for use in future sessions. To preserve the current filter configuration, click the Save Filter button on the Filter page. Type in a name for the filter, then click Save to save the filter and return to the Filter page. Loading a saved filter Once a filter configuration has been saved, you can retrieve it in later sessions by loading it back into your Gene Sorter. To load the saved filter settings, click the Load Filter button on the Filter page. Click on the name of the filter you wish to load, then click the Load button. Click the Submit button on the Filter page to apply the filter settings to the Gene Sorter. Viewing a list of saved filters To display a list of filter settings that you have saved, click the Save button on the Filter page. If you have any saved filters, the Gene Sorter will display an Existing Setups list that shows the filter names. To permanently remove a filter from the list, click on the name to highlight it, then click the Delete Existing Setup button. Displaying sequence and text-based output The Gene Sorter's graphical presentation of data facilitates the visual observation of relationships and patterns among the genes in the display. However, it is often useful to convert the data to a text-based format that can be easily saved to a file or loaded into another program, database, or spreadsheet for further analysis. The Gene Sorter provides a mechanism for saving the current display in a tab-delimited text file or showing a text-based view of the sequence underlying the current display. Creating text-based output To output the current Gene Sorter table as text, click the text button at the top of the page. The Gene Sorter will display each row of table data on a separate tab-delimited line. Viewing the underlying sequence To display the protein, mRNA, or genomic sequence underlying the current Gene Sorter table, click the sequence button at the top of the page. On the Get Sequence page, select the desired sequence configuration settings that you'd like, then click the Get Sequence button. The Gene Sorter will display a text-based list of FASTA format records for each gene displayed in the table. The FASTA records may be cut and pasted into Blat for further study. 
/goldenPath/help/ftp.html:Genome_Browser_Data_Downloads	Downloading data Rsync (recommended method) We recommend that you download data via rsync using the command line, especially for large files using the North American or European download servers. For example, when downloading files to your present directory (./), use an expression such as: rsync -a -P rsync://hgdownload.soe.ucsc.edu/gbdb/hg38/uniprot/unipAliTrembl.bb ./ rsync -a -P rsync://hgdownload-euro.soe.ucsc.edu/gbdb/hg38/uniprot/unipAliTrembl.bb ./ To download the entire directory (note the trailing slash), use an expression such as: rsync -a -P rsync://hgdownload.soe.ucsc.edu/gbdb/hg38/uniprot/ ./ rsync -a -P rsync://hgdownload-euro.soe.ucsc.edu/gbdb/hg38/uniprot/ ./ To obtain a file's location (URL) to use with rsync (rsync://hgdownload.soe.ucsc.edu/... or rsync://hgdownload-euro.soe.ucsc.edu/...), navigate in your browser to our FTP site at ftp://hgdownload.soe.ucsc.edu/ or our downloads page at http://hgdownload.soe.ucsc.edu/downloads.html, and look for your file of interest. To learn more about rsync's options, type "man rsync" on the command line. Downloading from a web browser If you are not comfortable using the command line, you can download your file via FTP in your browser at ftp://hgdownload.soe.ucsc.edu/goldenPath, or from our downloads page at http://hgdownload.soe.ucsc.edu/downloads.html. However, downloading via your browser will be very slow or may even time out for large files (i.e., bigBed, bigWig, BAM, VCF, etc.). Also, Safari does not support FTP inside the browser. When you enter a FTP URL in Safari, you may have to select "Guest" and click submit to log in before a FTP file system will open in a window on your desktop. FTP from the command line We do not encourage the use of FTP for downloading large data files. Rsync is a more efficient and convenient transport mechanism, and is therefore quicker and easier to use for downloading our data files. However, here are the command line steps for FTP, should you choose to use it: $ ftp hgdownload.soe.ucsc.edu Name: anonymous Password: <your email address> ftp> cd goldenPath ftp> cd <assembly name> (e.g., hg19) ftp> cd <data directory> (e.g., liftOver) The same ftp connection can also be made to our European server: $ ftp hgdownload-euro.soe.ucsc.edu To download multiple files from the UNIX ftp command line, use the "mget" command. You may want to use the prompt command to toggle the interactive mode if you do not want to be prompted for each file that you download. ftp> mget [filename1] [filename2] ... Or to download all the files in the directory: ftp> mget -a 
/goldenPath/help/chimera.html:Genome_Browser_Chimera_Interface	Interface to UCSF Chimera Download Chimera Configure for automatic launch in web browsers UCSF Chimera is an interactive molecular modeling system that supports 3-D visualization of protein structures.It runs as an application on the user's computer. The UCSC Genome Browser provides links to launch Chimera as a helper application for proteins with solved structures deposited in PDB. Protein residues with non-synonymous SNPs identified by LS-SNP PDB are annotated in Chimera by the UCSC Browser for genome assemblies with LS-SNP PDB entries (currently human). Links to Chimera are available in the "Protein Domain and Structure Information" section of the UCSC Genes details page when the gene is associated with a PDB structure. Residues with non-synonymous SNPs are colored gold and labeled with the dbSNP identifier. Additionally, for SNPs in LS-SNP PDB, links to Chimera are generated on the SNP details page with the residue associated with the selected SNP colored red, and the other non-synonymous SNP residues colored gold. Chimera version 1.3 or newer must be installed locally on your computer from the Download Chimera page. To automatically launch Chimera from one of these links, it must be configured as a web browser helper application as described in the Chimera Web Data page. IMPORTANT NOTE: The UCSC Genome Browser group is unable to provide support for installing or using UCSF Chimera. If you need help, please see the UCSF Chimera Documentation or inquire on the Chimera users discussion mailing list. 
/goldenPath/help/genomeEuro.html:Genome_Browser_Supported_Mirrors	Using a Supported Mirror Site The UCSC Genome Browser hosts the following official mirror sites: the European mirror (genome-euro), located at the Universität Bielefeld Center for Biotechnology in Bielefeld, Germany and the Asian mirror (genome-asia), located at the RIKEN Yokohama Campus, Japan. These mirror sites are administered by UCSC and are meant to be alternate, faster access points for those Browser users who are geographically closer to central Europe or Asia than to the western United States. When users navigate to the U.S. server home page (and conversely, when U.S. users navigate to the European or Asian server) and click the "Genomes" menu item, they will receive a notification at the top of the window that they have been redirected to the more geographically appropriate server. At this point, one of two things can happen: 1. The user may choose to remain on the redirected server and proceed normally. 2. The user may click the provided link back to the original server. When this occurs, a cookie is created in the user's Internet browser remembering this choice and in the future, no automatic redirection will occur. The user may also manually select a server at any time by mousing over the "Mirrors" menu item, illustrated below. Note that manually selecting a server will also create a cookie that will disable automatic redirection. Automatic redirection can be re-enabled by clearing cookies. [] Many thanks to our colleagues at the Universität Bielefeld Center for Biotechnology for hosting genome-euro, and at RIKEN for hosting genome-asia. About Saved Sessions and Custom Tracks on Official Mirror Sites When genome-euro was first activated in 2013, the genome-euro Saved Sessions database was an identical copy of the Saved Sessions database from the U.S. server, so Saved Session data were the same on the two servers, except for custom tracks. After the initial activation of genome-euro, however, the Saved Sessions between the two servers began to diverge. New Saved Sessions or modifications to existing Saved Sessions are NOT copied from one server to the other, so a new Saved Session created on either server will NOT be available on the other. This is also the case on genome-asia (activated in 2016). Saved Sessions can be moved from one server to the other by saving them to a file and reloading the file using "Save current settings to a local file:" and "Use settings from a local file:" features on the Sessions page Custom Track data, including those Custom Tracks belonging to Saved Sessions, are NOT copied from the U.S. server to mirror sites. Any Custom Tracks that exist on either the U.S. server or a mirror site will need to be manually downloaded via the Table Browser or otherwise manually recreated if users would like them to be present on the other. Once Custom Tracks are loaded onto a mirror site, a Saved Session may be re-saved under the same name to reconstitute the original session with all associated Custom Tracks. If you would like your custom data to be easily available on an official mirror site and the UCSC Genome Browser site, we highly recommend creating a Track Hub. A track hub has the benefit of being hosted on your site allowing more control, ensuring data persistence, minimizing upload time to the browser, simplifying sharing of data, and optimizing track functionality. 
/goldenPath/help/bedgraph.html:Genome_Browser_bedGraph_Track_Format	BedGraph Track Format The bedGraph format allows display of continuous-valued data in track format. This display type is useful for probability scores and transcriptome data. This track type is similar to the wiggle (WIG) format, but unlike the wiggle format, data exported in the bedGraph format are preserved in their original state. This can be seen on export using the table browser. For more details on data compression in wiggle tracks see the notes section of the wiggle track description page. If you have a very large data set and you would like to keep it on your own server, you should use the bigWig data format. In fact, an attempt to load a bedGraph custom track over 50,000,000 lines will result in an error message, but can be addressed by turning the bedGraph into a bigWig (see Example 3). Note that bedGraph files cannot easily be converted to wiggle files; converting bedGraph to bigWig and using bigWigToWig will return the original bedGraph file. General Structure The bedGraph format is line-oriented. BedGraph data are preceded by a track definition line, which adds a number of options for controlling the default display of this track. Following the track definition line are the track data in four column BED format: chromA chromStartA chromEndA dataValueA chromB chromStartB chromEndB dataValueB Parameters for bedGraph track definition lines All options are placed in a single line separated by spaces: track type=bedGraph name=track_label description=center_label visibility=display_mode color=r,g,b altColor=r,g,b priority=priority autoScale=on|off alwaysZero=on|off gridDefault=on|off maxHeightPixels=max:default:min graphType=bar|points viewLimits=lower:upper yLineMark=real-value yLineOnOff=on|off windowingFunction=maximum|mean|minimum smoothingWindow=off|2-16 Note: if you copy/paste the above example, you must remove the line breaks. The track type is REQUIRED, and must be bedGraph: type=bedGraph The remaining values are OPTIONAL. The wiggle documentation contains details on these options. A functional description of these options can be seen in the track configuration description. (Custom tracks do not have interactive configuration options.) Data Values BedGraph track data values can be integer or real, positive or negative values. The chromosome coordinates are zero-based, half-open. This means that the first chromosome position is 0, and the last position in a chromosome of length N would be N - 1. The positions listed in the input data must be in numerical order, and only the specified positions will be graphed. bedGraph format has four columns of data: chrom chromStart chromEnd dataValue Example This example specifies 9 separate data points in three tracks on chr19 in the region 49,302,001 to 49,304,701. To view this example as a custom track in the Genome Browser, copy the text and paste it into the browser annotation track text box. browser position chr19:49302001-49304701 browser hide all browser pack refGene encodeRegions browser full altGraph # 300 base wide bar graph, autoScale is on by default == graphing # limits will dynamically change to always show full range of data # in viewing window, priority = 20 positions this as the second graph # Note, zero-relative, half-open coordinate system in use for bedGraph format track type=bedGraph name="BedGraph Format" description="BedGraph format" visibility=full color=200,100,0 altColor=0,100,200 priority=20 chr19 49302000 49302300 -1.0 chr19 49302300 49302600 -0.75 chr19 49302600 49302900 -0.50 chr19 49302900 49303200 -0.25 chr19 49303200 49303500 0.0 chr19 49303500 49303800 0.25 chr19 49303800 49304100 0.50 chr19 49304100 49304400 0.75 chr19 49304400 49304700 1.00 Note: The above example is a custom track that includes a track type= line that is specific for loading the data in the browser. This line will cause a raw bedGraph data file to fail validation by other tools, such as validateFiles, outside of the browser. 
/goldenPath/help/gbic.html:GBIC	Genome Browser in the Cloud User's Guide Contents What is Genome Browser in the Cloud? Quick Start Instructions How does the GBiC program work? GBiC Commands All GBiC options Credits What is Genome Browser in the Cloud? The Genome Browser in the Cloud (GBiC) program is a convenient tool that automates the setup of a UCSC Genome Browser mirror. The GBiC program is for users who want to set up a full mirror of the UCSC Genome Browser on their server/cloud instance, rather than using Genome Browser in a Box (GBIB) or our public website. Please see the Installation of a UCSC Genome Browser on a local machine (mirror) page for a summary of installation options, including the pros and cons of using a mirror installation via the GBiC program vs. using GBiB. GBiC can also be used as a Docker container. See our Docker help page for more information. The program works by setting up MySQL (MariaDB), Apache, and Ghostscript, and then copying the Genome Browser CGIs onto the machine under /usr/local/apache/. Because it also deactivates the default Apache htdocs/cgi folders, it is best run on a new machine, or at least a host that is not already used as a web server. The tool can also download full or partial assembly databases, update the Genome Browser CGIs, and remove temporary files (aka “trash cleaning”). The GBiC program has been tested and confirmed to work with Ubuntu 18/20/22/24 LTS, Rocky 9.5, and Fedora 30/35/41. It has also been tested on virtual machines in Amazon EC2 (Centos) and Microsoft Azure (Ubuntu). If you want to load data on the fly from UCSC, you need to select the data centers “US West (N. California)” (Amazon) or “West US” (Microsoft) for best performance. Other data centers (e.g. East Coast) will require a local copy of the genome assembly, which requires 2TB-7TB of storage for the hg19 assembly. Note that if you mirror multiple assemblies, this may in rare cases exceed 16TB, the current maximum size of a single Amazon EBS volume. In these cases, you may need to use RAID striping of multiple EBS volumes, or use Amazon EFS for the /gbdb files and Amazon Aurora for the MySQL tables. We have had one report from a user that the MySQL performance is better when served from Aurora instead of being served from the same EC2 instance; the cost may be higher in Aurora, depending on the type of usage. Quick Start Instructions Download the GBiC program from the UCSC Genome Browser store. Run the program as root, like this: sudo bash browserSetup.sh install The install command downloads and configures Apache, MySQL (MariaDB) and Ghostscript, copies the Genome Browser CGIs, and configures the mirror to load data remotely from UCSC. The install command must be run before any other command is used. For mirror-specific help, please contact the Mirror Forum as listed on our contact page. For an installation demonstration, see the Genome Browser in the Cloud (GBiC) Introduction video: How does the GBiC program work? The GBiC program downloads the Genome Browser CGIs and sets up the central MySQL (MariaDB) database. All potentially destructive steps require confirmation by the user (unless the -b batch mode option is specified). In particular, MySQL (MariaDB) and Apache are installed and set up with the right package manager (yum or apt-get). A default random password is set for the MySQL (MariaDB) root user and added to the ~/.my.cnf file of the Unix root account. If you have already set up MySQL (MariaDB), you must create the ~/.my.cnf file. The program will detect this and create a template file for you. The program also performs some minor tasks such as placing symlinks, detecting MariaDB, deactivating SELinux, finding the correct path for your Apache install and adapting the MySQL (MariaDB) socket config. This will result in a Genome Browser accessible on localhost that loads its data through genome-mysql.soe.ucsc.edu:3306 and hgdownload.soe.ucsc.edu:80. If your geographic location is not on the US West Coast, the performance will be too slow for normal use, though sufficient to test that the setup is functional. A special MySQL (MariaDB) server is set up in Germany for users in Europe. You can change the /usr/local/apache/cgi-bin/hg.conf genome-mysql.soe.ucsc.edu lines to genome-euro-mysql.soe.ucsc.edu in order to get better performance. You can then use the program to download assemblies of interest to your local Genome Browser, which will result in performance at least as fast as the UCSC site. Network requirements Your network firewall must allow outgoing connections to the following servers and ports: - MySQL (MariaDB) connections, used to load tracks not local to your computer: - US server: Port 3306 on genome-mysql.soe.ucsc.edu (128.114.119.174) - European server: Port 3306 on genome-euro-mysql.soe.ucsc.edu (129.70.40.120) - Rsync, used to download track data: - US server: TCP port 873 on hgdownload.soe.ucsc.edu (128.114.119.163) - European server: TCP port 873 on hgdownload-euro.soe.ucsc.edu (129.70.40.99) - Download HTML descriptions on the fly: - US server: TCP port 80 on hgdownload.soe.ucsc.edu (128.114.119.163) - European server: TCP port 80 on hgdownload-euro.soe.ucsc.edu (129.70.40.99) Root file system too small for all data If you need to move data to another partition because the root file system is too small for all of the assembly's data, the following steps will help complete the installation. First, do a minimal installation with the browserSetup.sh script as described below, using just the “install” argument. Then make symlinks to the directory that will contain the data, e.g. if your biggest filesystem is called “/big”: sudo mv /var/lib/mysql /big/ sudo mv /gbdb /big/ sudo ln -s /big/mysql /var/lib/mysql sudo ln -s /big/gbdb /gbdb Then use the “mirror” or “minimal” arguments to browserSetup.sh to rsync over the majority of the data. GBiC Commands The first argument of the program is called command in the following section of this document. The first command that you will need is install, which installs the Genome Browser dependencies, binary files and basic MySQL (MariaDB) infrastructure: sudo bash browserSetup.sh install There are a number of options supported by the GBiC program. In all cases, options must be specified before the command. The following example correctly specifies the batch mode option to the program: sudo bash browserSetup.sh -b install To improve the performance of your Genome Browser, the program accepts the command minimal. It will download the minimal tables required for reasonable performance from places in the US and possibly others, e.g., from Japan. Call it like this to trade space for performance and download a few of the most used MariaDB tables for hg38: sudo bash browserSetup.sh minimal hg38 If the Genome Browser is still too slow, you will have to mirror all tables of a genome assembly. By default, rsync is used for the download. Alternatively you can use UDR, a UDP-based fast transfer protocol (option: -u). sudo bash browserSetup.sh -u mirror hg38 A successful run of mirror will also cut the connection to UCSC: no tables or files are downloaded on-the-fly anymore from the UCSC servers. To change the remote on-the-fly loading, specify the option -o (offline) or -f (on-the-fly). If you are planning to keep sensitive data on your mirror, you will want to disable on-the-fly loading, like so: sudo bash browserSetup.sh -o If you notice that BLAT no longer works for your assembly while in offline-mode, you may have to update your GBiC mirror to restore functionality. Patch sequences are often added to assemblies which may result in errors if your CGIs and 2bit files have not been updated to the latest version. The full assembly download for hg19 is >7TB. Limit this to 2TB or less with the -t option: sudo bash browserSetup.sh -t noEncode mirror hg19 For a full list of -t options, see the All GBiC options section or run the program with no arguments. To update all CGIs and fully mirrored assemblies, call the tool with the update parameter like this: sudo bash browserSetup.sh update The update parameter can also be used to update the data for a single assembly by providing the UCSC assembly name (e.g. rn6, bosTau6, equCab3): sudo bash browserSetup.sh update bosTau9 Minimal mirror sites (those that have partially mirrored an assembly) should not use the update command, but rather just rerun the minimal command, so that only the minimal tables are updated. For instance, if you have partially mirrored the hg19 and hg38 databases, you may want to add this command to your crontab, perhaps running it every day, to keep your local tables in sync with those at UCSC: sudo bash browserSetup.sh minimal hg19 hg38 To update only the Genome Browser software and not the data, use the cgiUpdate command: sudo bash browserSetup.sh cgiUpdate Software may break or not work correctly if the necessary data is not available. Thus in most circumstances, we recommend you use the mirror, update, or minimal commands instead of cgiUpdate. You will also want to add a cleaning command to your crontab to remove the temporary files that are created during normal Genome Browser usage. These accumulate in /usr/local/apache/trash and can quickly consume significant space. A command like this should be added to your crontab file: sudo bash browserSetup.sh clean If you find that you need the Kent command line utilities in addition to the Genome Browser, the addTools command will install all the utilities into /usr/local/bin: sudo bash browserSetup.sh addTools A majority of these utilities require an .hg.conf file in the users home directory. For an example of a minimal .hg.conf file, click here. If you find a bug, or if your Linux distribution is not supported, please contact genome-mirror@soe.ucsc.edu. More details about the Genome Browser installation are available here. All GBiC options Here is the full listing of commands and options supported by the GBiC program: browserSetup.sh [options] [command] [assemblyList] - UCSC genome browser install script command is one of: install - install the genome browser on this machine. This is usually required before any other commands are run. minimal - download only a minimal set of tables. Missing tables are downloaded on-the-fly from UCSC. mirror - download a full assembly (also see the -t option below). No data is downloaded on-the-fly from UCSC. update - update the genome browser software and data, updates all tables of an assembly, like "mirror" cgiUpdate - update only the genome browser software, not the data. Not recommended, see documentation. clean - remove temporary files of the genome browser older than one day, but do not delete any uploaded custom tracks addTools - copy the UCSC User Tools, e.g. blat, featureBits, overlapSelect, bedToBigBed, pslCDnaFilter, twoBitToFa, gff3ToGenePred, bedSort, ... to /usr/local/bin parameters for 'minimal', 'mirror' and 'update': <assemblyList> - download MySQL + /gbdb files for a space-separated list of genomes examples: bash browserSetup.sh install - install Genome Browser, do not download any genome assembly, switch to on-the-fly mode (see the -f option) bash browserSetup.sh minimal hg19 - download only the minimal tables for the hg19 assembly bash browserSetup.sh mirror hg19 mm9 - download hg19 and mm9, switch to offline mode (see the -o option) bash browserSetup.sh mirror -t noEncode hg19 - install Genome Browser, download hg19 but no ENCODE tables and switch to offline mode (see the -o option) bash browserSetup.sh update hg19 - update all data and all tables of the hg19 assembly (in total 7TB) bash browserSetup.sh cgiUpdate - update the Genome Browser CGI programs bash browserSetup.sh clean - remove temporary files older than one day All options have to precede the command. options: -a - use alternative download server at SDSC -b - batch mode, do not prompt for key presses -t - only download track selection, requires a value. This option is only useful for Human/Mouse assemblies. Download only certain tracks, possible values: noEncode = do not download any tables with the wgEncode prefix, except Gencode genes, saves 4TB/7TB for hg19 bestEncode = our ENCODE recommendation, all summary tracks, saves 2TB/7TB for hg19 main = only RefSeq/Gencode genes and common SNPs, total 5GB for hg19 -u - use UDR (fast UDP) file transfers for the download. Requires at least one open UDP incoming port 9000-9100. (UDR is not available for Mac OSX) This option will download a udr binary to /usr/local/bin -o - switch to offline-mode. Remove all statements from hg.conf that allow loading data on-the-fly from the UCSC download server. Requires that you have downloaded at least one assembly, using the '"download"' command, not the '"mirror"' command. -f - switch to on-the-fly mode. Change hg.conf to allow loading data through the internet, if it is not available locally. The default mode unless an assembly has been provided during install -h - this help message Credits - Max Haeussler for writing the program. - Christopher Lee for testing and QA. - Daniel Vera (bio.fsu.edu) for his RHEL install notes. - Bruce O'Neill, Malcolm Cook for feedback. 
/goldenPath/help/multiRegionHelp.html:Genome_Browser_Multi-Region	Multi-Region Display Help Contents Introduction Getting started with Multi-Region View About the Multi-Region modes - Exon only - Gene only - Custom regions - Haplotype view Video demonstrations of Multi-Region modes - Exon only - Custom regions - Haplotype view Examples Search the Genome Browser help pages:   Questions and feedback are welcome. Introduction The multi-region display allows users to "slice" their track-viewing experience into a variety of different modes that focus the display on certain features: exon-only, gene-only, or user-defined BED coordinates. Only the portions of track annotations that fall within these displayed regions are shown; extraneous intergenic, intronic and otherwise unwanted regions are hidden from view. For human assemblies hg17 and later, the multi-region view also supports the replacement of a section of the reference genome with an alternate haplotype chromosome. This allows the user to view annotations upstream and downstream of the haplotype sequence, and visualize the haplotype in the general context of the reference chromosome. Using the multi-region display, one can: - Show all SNPs that fall within exons (exon-only view). - Show only the genes on a chromosome (gene-only view). - Display the same chromosomal position on either side of a gene in order to model a repeat region (using custom regions in BED format). - See how an alternate haplotype fits onto its chromosome (alternate haplotype view). Getting started With Multi-Region view Follow these steps to activate and configure multi-region view in the Genome Browser. Step 1. Pick a genome assembly and region Open the Genome Browser tracks display to a region of interest within a standard genome assembly, or open a custom region within an assembly hub. Step 2. Open the multi-region view configuration window Click multi-region under the tracks display to open the multi-region configuration window. [Multi-Region button] Step 3. Configure the multi-region view By default, the Exit multi-region mode button is selected. This button returns the display to the traditional view. [Configure Multi-Region view] To activate a multi-region view, click one of these options: - Show exons using -- display only the exonic regions (keyboard shortcut: "e v") - Show genes using -- display only those regions with gene annotations in the specified genes track - Enter custom regions as BED, or a URL to them -- display only those regions entered into the text box in BED format, or listed in a specified BED file - Show one alternate haplotype -- display the specified alternate haplotype in place on the reference chromosome (human assemblies hg17 and later only) - (Optional) Select Highlight alternating regions in multi-region view to color alternating slices in blue. Click submit to save your changes and return to the Genome Browser tracks display. Here is a view of the Genome Browser with the "Show exons" and "Highlight alternating regions" options selected: [Exon-only screenshot] Navigation and display With a few exceptions, most browser navigation and display features function the same in both multi-region and single-chromosome modes, including zooming in and out, reversing and resizing the view, highlighting and zooming to a selected region, and track search. Searching and zooming conventions differ when displaying custom BED regions. See About Custom Regions for more information. About the Multi-Region modes Exon-only mode The exon-only view shows only those bases and annotations that fall within an exon or padding region. Bases and alignments within intronic and intergenic regions are removed from the display. The exon-only view "virtual chromosome" is constructed of all the exons present in the gene track specified in the multi-region configuration window. Exon "slices" are visually separated by blocks of intronic or intergenic bases, the size of which are set by the padding field. This designates the padding on both sides of an exon, and therefore is additive, e.g., a padding size of 6 will result in a 12-base space between exon slices: [Multi-Region padding] If the gene track used to compute the slicing is a super track or contains other transcripts, only the longest exon will be used to determine slicing, regardless of whether or not they are in the same transcript. See Example 1 for more information. Video Gene-only mode The gene-only view displays the bases and annotations that fall within the coordinates of a gene, but hides all of the intergenic bases except those used for padding (Example 2). As with exon-only mode, the padding field sets the number of intergenic bases displayed to visually separate each gene slice. Because genes and alternate transcripts often overlap by a few bases, the windows created by the gene-only (and exon-only) mode may sometimes merge together. Reducing the number of padding bases may increase the number of alternate regions by allowing less intergenic space between genes. However, this overlap is a common feature for most of the Genome Browser gene sets, and should be no cause for concern. Custom regions mode The custom regions mode allows user-specific modeling of genomic regions by either entering them into the text box, or linking to a BED file. Typically when a BED file is loaded into the browser, the entries are displayed in the tracks window in order of chromosomal location, regardless of the order in the file. When the multi-region view option is selected, the BED file entries are displayed in the same order they appear in the file. This feature can be used to manipulate the display of other annotations in the browser window, for example to duplicate a display region, reorder the exon display within a gene, or display regions from 2 different chromosomes side-by-side. Example 3 demonstrates the importance of ordering entries within a file. Although all types of BED files are supported, only the information from BED 3 or BED 12 files is relevant to the display. Like BED custom tracks, comments are supported in the BED files used by multi-region mode. Comments are included at the top of the BED file, preceded by "#". The supported comments are: - #database: The genome assembly database for which the BED file is intended. If the file is loaded onto a different assembly, the custom regions mode will not work. - #shortDesc: (Optional) A short description of the custom region, displayed next to the chromosome ideogram. - #padding: Sets the size of the padding on either side of a multi-region boundary in the browser tracks window. Here is an example of a BED3 file with comments: #database hg38 #shortDesc Example of a short description #padding 6 chr1 pos1 pos2 chr2 pos1 pos2 The Genome Browser search mechanism functions differently when using custom BED regions in multi-region view. When searching for a term or position, the browser first searches for the term within the BED-defined regions. If the item is found, the browser display will shift to that location. If the term is not found within the BED-defined regions, but is found elsewhere, the browser will display a message asking if you would like to return to a default view at that location. If you prefer to return to your custom region instead, click OK. Otherwise, click "here" to return to the default view. [Multi-Region search redirect] If the term is not found at all in the assembly, an error message will display. When scrolling or zooming while viewing regions from a custom BED URL, the range is restricted to the BED-defined regions. Although you can still zoom in to single base resolution, the outward zoom is limited to the extent of the defined regions. Scrolling upstream and downstream is also limited to the BED-defined range. Video Haplotype mode An additional viewing mode is available on human assemblies hg17 and later: showing an alternative haplotype placed in its chromosomal context (Example 5). The haplotype is specified by entering its ID in the form chr_name_alt in the configuration window. To view the list of haplotype IDs for a given assembly, go to http://hgdownload.soe.ucsc.edu/goldenPath/$db/bigZips/$db .chrom.sizes, where $db is the human assembly name (e.g., hg38, hg19, hg18 or hg17). Alternatively, haplotypes can be viewed using the Alt Map... track in the Mapping and Sequencing group. Open the track and navigate to the haplotype of interest in the track window. Click on the haplotype to display its details page, then click the link Show this alternate haplotype placed on its chromosome. Video Videos Video demonstration of Exon-only mode []       Visit our Video Page.       Visit our YouTube channel. Video demonstration of Custom Regions []       Visit our Video Page.       Visit our YouTube channel. Video demonstration of Haplotype Substitution using Multi-Region []       Visit our Video Page.       Visit our YouTube channel. Examples Example 1. Exon-only mode This screenshot shows the exon-only view for the SOD1 gene in human assembly hg38. The third "exon" from the left falls within the intron of the actual SOD1 gene, but is shown as an exon because it is part of the GENCODE v22 super-track. [Screenshot of exon-only mode] Click here to view this example in the Genome Browser. To quickly exit multi-region mode and return to the default single chromosome view, use the keyboard shortcut "d v". Example 2. Gene-only mode This example illustrates how intergenic regions are removed from the default browser tracks view in gene-only mode. The displayed region of the X chromosome on the human hg19 assembly consists of three genes (DKC1, MPP1, and F8) with varying amounts of intergenic space separating them. The first screenshot shows the standard browser display of the gene track in "single chromosome" view. Compare it with the second screenshot, which shows the same region in gene-only mode with the padding set to 6. (Click here to view this example in the Genome Browser.) [Genome Browser default gene track] [Multi-Region gene-only view] To quickly exit multi-region mode and return to the default single chromosome view, use the keyboard shortcut "d v". Example 3. Custom regions from BED file Here is an example BED file for mm10 that contains a list of exon positions for the Pax9 gene. If this file is loaded into the browser as a custom track without turning on multi-region mode, the blocks display in order of genomic position, not in the order they are specified in the file: [Genome Browser default gene display] However, if multi-region mode is turned on after loading a custom track, the blocks display in the order they are listed in the BED file, rather than in genomic position order. This in turn affects the display order of other features in this view, in this case the display order of the exons in Pax9. This feature can be used to manipulate the display of other annotations, for example to allow the visualization of centromeric regions or viewing the same gene twice within the same window to compare annotations in a separate region. [Multi-Region bed custom output] Click here to view this example in the Genome Browser. To quickly exit multi-region mode and return to the default single chromosome view, use the keyboard shortcut "d v". Example 4. BED12 The following BED file utilizes BED12 format to display two regions from two different chromosomes, chromosome 22 and chromosome 21, side-by-side in the browser tracks window, along with human genes POTEH, CECR2, USP25 and CHODL. chr22 15690026 17558149 chr22 960 + 15690026 17558149 0 2 31497,81057, 0,1787066 chr21 15730025 18267373 chr21 960 + 15730025 18267373 0 2 148887,22540, 0,2514808 [Multi-Region BED12 example] The multi-region display utilizes the data in the blockStarts and blockSizes fields of the BED file to draw its regions, allowing for even more customization of the Genome Browser display. Click here to view this example in the Genome Browser. To quickly exit multi-region mode and return to the default single chromosome view, use the keyboard shortcut "d v". Example 5. Haplotype This example shows how the human hg38 haplotype chr1_KI270762v1_alt fits onto chromosome 1. Note that annotations from the standard browser display do not extend into or out of the haplotype region, since haplotypes are annotated separately from regular chromosomes. [Multi-Region haplotype view] Click here to view this example in the Genome Browser. To quickly exit multi-region mode and return to the default single chromosome view, use the keyboard shortcut "d v". 
/goldenPath/help/multiView.html:Genome_Browser_Multi-View	Configuring Multi-View Tracks The drop-down menus in the Select views section control the kind of data tracks that will be displayed, such as "peaks" or "signal". To configure the display of the tracks for each data type, click on the name of the view next to the drop-down menu. Note that when a view has been set to hide in this section, the corresponding subtracks will have grayed-out checkboxes in the List subtracks section below. Selecting subtracks The Select subtracks section is a grid of checkboxes that provides a quick way to select a set of subtracks according to some attribute, such as "cell line" or "antibody". Note that selections made using these boxes reset any selections made on the individual subtracks in the List subtracks section below. Some tracks allow you to Select subtracks further by additional categories such as "treatment". Click on a menu to see all of the available choices. Choices that are not compatible with the current set of selected subtracks may be grayed out. Click on one or more choices to select them. The List subtracks section allows you to select and deselect individual subtracks. This section provides the most refined control of the tracks displayed. Selections made here do not change the selections made in the Select views or Select subtracks sections above.Note that if the subtrack list is restricted to "only selected/visible" then some available subtracks may be hidden from the list. If the list of subtracks is long, this may be useful. Sorting subtracks The order of subtracks in the List subtracks section determines the display position on the main Genome Browser display page. Tracks can be sorted by clicking on the attribute links at the top of the column (if any). The arrows next to the attribute links indicate which direction the columns are being sorted. The numbers next to the arrows indicate the priority of the sort for each attribute. Additionally, some subtracks can be ordered by "drag and drop": click and hold a subtrack checkbox in the List subtracks section and release it in a new position. Note that tracks may be reordered by "drag and drop" in the main Genome Browser display. Those changes will not be reflected in this list. Configuring individual subtracks Subtracks can have display and configuration settings that are distinct from the set they belong to. Click on the display mode (hide, dense, etc.) of a subtrack shown in the List subtracks to change it or click on the wrench icon ([Wrench Icon]) to change the configuration settings for an individual subtrack. Individual subtrack settings will be overwritten anytime the set of tracks is configured as a whole. More details To learn more about the underlying data for a subtrack, press the down arrow ([Down arrow]) next to the subtrack name in the List subtracks section. This section will expand to display information such as the name of the data contributor, the cell line and antibody used in the experiment, the data submission date and more. To see a full description of the underlying table, press the "schema" link. Selecting filter categories Some tracks allow you to control the items that appear in displayed subtracks using Filter items by categories. Click on a filter menu to see all of the available item types. Click on one or more item types to select them. Selecting highlighting categories Some tracks allow you to highlight the items by various attributes using Highlight items by categories. Click on a highlight menu to see all of the available item types. Click on one or more item types to select them. 
/goldenPath/help/hgHicTrackHelp.html:Genome_Browser_hic_Tracks	Configuring hic tracks Genome Browser hic tracks display heatmaps of chromatin folding data and may be configured in several ways to highlight different aspects of the data. By default, hic tracks will use auto-scale options for resolution and score coloring. The following section describes configuration settings available to hic tracks. For more information on setting up and uploading hic data, click the link on hic custom track creation. Configuring the display mode There are three display methods available for Hi-C heatmaps: square, triangle, and arc. [hic draw modes] Square mode provides a traditional Hi-C display in which chromosome positions are mapped along the top-left-to-bottom-right diagonal. Interaction values are plotted symetrically on both sides of that diagonal to form a square. The upper-left corner of the square corresponds to the left-most position of the window in view, while the bottom-right corner corresponds to the right-most position of the window. The color shade at any point within the square shows the proximity score for two genomic regions: the region where a vertical line drawn from that point intersects with the diagonal, and the region where a horizontal line from that point intersects with the diagonal. A point directly on the diagonal shows the score for how proximal a region is to itself (scores on the diagonal are usually quite high unless no data are available). A point at the extreme bottom left of the square shows the score for how proximal the left-most position within the window is to the right-most position within the window. In triangle mode, the display is quite similar to square except that only the top half of the square is drawn (eliminating the redundancy), and the image is rotated so that the diagonal of the square now lies on the horizontal axis. This display consumes less vertical space in the image, although it may be more difficult to ascertain exactly which positions correspond to a point within the triangle. In arc mode, simple arcs are drawn between the centers of interacting regions. The color of each arc corresponds to the proximity score. Self-interactions are not displayed. Score normalization settings Score values for this type of display correspond to how close two genomic regions are in 3D space. A high score indicates more links were formed between them in the experiment, which suggests that the regions are near to each other. A low score suggests that the regions are farther apart. High scores are displayed with a more intense color value; low scores are displayed in paler shades. There are four score values available in this display: NONE, VC, VC_SQRT, and KR. NONE provides raw, un-normalized counts for the number of interactions between regions. VC, or Vanilla Coverage, normalization (Lieberman-Aiden et al., 2009) and the VC_SQRT variant normalize these count values based on the overall count values for each of the two interacting regions. Knight-Ruiz, or KR, matrix balancing (Knight and Ruiz, 2013) provides an alternative normalization method where the row and column sums of the contact matrix equal 1. Color intensity in the heatmap goes up to indicate higher scores, but eventually saturates at a maximum beyond which all scores share the same color intensity. The value of this maximum score for saturation can be set manually by un-checking the "Auto-scale" box. When the "Auto-scale" box is checked, it automatically sets the saturation maximum to be double (2x) the median score in the current display window. Resolution settings The resolution for each track is measured in base pairs and represents the size of the bins into which proximity data are gathered. The list of available resolutions ranges from 1kb to 10Mb. There is also an "Auto" setting, which attempts to use the coarsest resolution that still displays at least 500 bins in the current window. When you have finished making your configuration changes, click the Submit button to return to the annotation track display page. References Knight P, Ruiz D. A fast algorithm for matrix balancing. IMA J Numer Anal. 2013 Jul;33(3):1029-1047. Lieberman-Aiden E, van Berkum NL, Williams L, Imakaev M, Ragoczy T, Telling A, Amit I, Lajoie BR, Sabo PJ, Dorschner MO et al. Comprehensive mapping of long-range interactions reveals folding principles of the human genome. Science. 2009 Oct 9;326(5950):289-93. PMID: 19815776; PMC: PMC2858594 
/goldenPath/help/geneSearchBox.html:Genome_Browser_Gene_Search	How to use the Search Box to find a gene The search box accepts input in several formats. The search box has a gene name auto-complete feature that suggests matches and takes users directly to the position of the UCSC/Known Genes or Refseq record associated with the gene, bypassing a search of the entire database. [] To use this functionality, start typing the gene name into the search box; it will suggest gene names after 2 or more characters are entered. These suggestions can be clicked, bringing you immediately to the position of the item you clicked. Alternately, if you finish typing and hit "Go" or "Enter", this will bring up a full list of up to 500 search results. 
/goldenPath/help/hubQuickStartGroups.html:Track_Hub_Groups_Quick_Start	Quick Start Guide to Organizing Track Hubs into Groupings Track hubs allow for displaying many tracks, therefore organizing your tracks using grouping settings will help your users find related information. Below is a basic example hub illustrating the use of container multiWig, compositeTrack on, and superTrack on lines. STEP 1: In a publicly-accessible directory, copy the hub.txt, genomes.txt, trackDb.txt, and examplePage.html files using the following command: wget -r --no-parent --reject "index.html*" -nH --cut-dirs=3 http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubGroupings/ Alternatively, if you do not have wget installed, use curl: curl -O http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubGroupings/hub.txt curl -O http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubGroupings/genomes.txt mkdir hg19 cd hg19 curl -O http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubGroupings/hg19/trackDb.txt curl -O http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubGroupings/hg19/examplePage.html If you do not have curl, you can use a text editor and directly recreate the above three files. Note: there is now a useOneFile on hub setting that allows the hub properties to be specified in a single file. More information about this setting can be found on the Genome Browser User Guide. STEP 2: Paste your hub.txt link (http://yourURL/hub.txt) into the Connected Hubs tab of the Track Data Hubs page, then click the "Genome Browser" link from the top bar. Alternatively build a URL that will directly load your hub in hgTracks: http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg19&hubUrl=http://yourURL/hub.txt The URL should work the same as using the original data just copied: http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg19&hubUrl=http://genome.ucsc.edu/goldenPath/help/examples/hubExamples/hubGroupings/hub.txt STEP 3: Congratulations! Your hub should display! If you are having problems, be sure all your files and the hg19 directory are publicly accessible. For hubs to work, your server must also accept byte-ranges. You can check using the following command to verify "Accept-Ranges: bytes" displays: curl -I http://yourURL/hub.txt The three types of track groupings Now that you have the hub copied from above, start to edit some of the trackDb.txt settings to understand how they work. Read more about trackDb settings in the definition document. Note that the Browser waits 5 minutes before checking for any changes to these files. When editing hub.txt, genomes.txt, trackDb.txt, and related hub files shorten this delay by adding udcTimeout=1 to your URL. For more information, please see the Debugging and Updating Track Hubs section of the Track Hub User Guide. For more detailed instructions on setting up a hub, please see the Setting Up Your Own Track Hub section of the Track Hub User Guide. - Understanding multiWig tracks - Understanding composite tracks - Understanding supertracks Resources - Track Hub User Guide - Track Database (trackDb) Definition Document - Assembly Hubs Wiki - Public Hub Guidelines - Basic Hub Quick Start Guide - Assembly Hub Quick Start Guide Understanding multiWig tracks track multiWigUniqueTrackName type bigWig container multiWig aggregate transparentOverlay showSubtrackColorOnUi on maxHeightPixels 500:100:8 ... track uniqueNameWithoutSpaces type bigWig parent multiWigUniqueTrackName color 255,0,0 A multiWig starts with a few related bigWig files that you want to display together. The container multiWig line allows for this track to be later referenced as parent multiWigUniqueTrackName in each of the related bigWig files. The aggregate transparentOverlay line defines the way the multiWigs should appear with options being transparentOverlay/stacked/solidOverlay. The showSubtrackColorOnUi on line shows the track colors on the track setting page and the maxHeightPixels 500:100:8 sets the maximum (500), default (100), and minimum (8) pixel heights for the track. Read all about multiWigs here. See an example trackDb.txt. Understanding composite tracks track uniqueCompositeTrackName compositeTrack on ... track uniqueNameWithoutSpaces parent uniqueCompositeTrackName on ... track newUniqueNameWithoutSpaces parent uniqueCompositeTrackName off A composite track groups together related tracks, usually but not necessarily of a similar type, that you want to display together (referred to as "subtracks"). If you want to organize tracks into a hierarchy and there is a single level of grouping, use a composite. For example, you could group together called variants or ChIP-seq peaks with their underlying BAM reads or sequencing coverage. The compositeTrack on line defines the parent track that will be later referenced as parent uniqueCompositeTrackName off in each subtrack's stanza. Either "on" or "off" can be used to set a subtrack to be displayed or not displayed by default. Composite tracks can be broken apart further to group very similar tracks with the trackDb use of subGroups and views, not demonstrated here. Read all about composite tracks here. See an example trackDb.txt. Understanding supertracks track uniqueSuperTrackName superTrack on show ... track uniqueNameWithoutSpaces parent uniqueSuperTrackName ... track newUniqueNameWithoutSpaces parent uniqueSuperTrackName ... track uniqueCompositeTrackNameInSuperTrack compositeTrack on parent uniqueSuperTrackName ... track uniqueNameWithoutSpaces parent uniqueCompositeTrackNameInSuperTrack on ... track newUniqueNameWithoutSpaces parent uniqueCompositeTrackNameInSuperTrack off A supertrack groups together different types of tracks - typically composites - in a high level folder. Use a supertrack if you need a second layer of hierarchy after composites. For example, you could have a composite with RNA-seq results and a composite with ChIP-Seq results grouped together into a supertrack describing a cell line. Supertracks contain composite tracks or container multiWigs, but not vice versa. The superTrack on show line allows for this track to be later referenced as parent uniqueSuperTrackName in each of the children subtracks (note how it is only required for direct children, and not for subtracks contained in a composite inside the supertrack -more below). The "show" is optional and sets the supertrack to display by default. It may help to think of the original declaring supertrack stanza as a light switch that by default is off, and can be flipped on by adding show. All tracks that claim membership to the supertrack can then set their own visibilities in lower stanzas by declaring settings such as by having the parent line and a separate visibility dense line. If no visibility setting is defined for a track, the default setting of hide is assigned. This can cause confusion if one mistakenly tries to set visibilities only at the top parent supertack stanza and leaves out visibility declarations for all children. Also do not confuse the parent line with how it is used in composites. For example, in supertracks DO NOT try something like ~~parent uniqueSuperTrackName [off/on]~~, where the [off/on] will only work with composite tracks. In the above nested example you do see track uniqueNameWithoutSpaces with a setting line specific only to the fact it is a child of a composite, parent uniqueCompositeTrackNameInSuperTrack on. The parent of the child composite track is in turn is a child to a supertrack declared by parent uniqueSuperTrackName. Read all about supertracks here. See an example trackDb.txt. 
/goldenPath/help/mirrorManual.html:Genome_Browser_Manual_Installation	Manual installation of the UCSC Genome Browser on a Unix server Contents Overview of the Genome Browser directories and databases Software Requirements Hardware and disk space requirements Installing the UCSC Genome browser MariaDB Setup Local Git repository (aka: "the source tree") Adding your own track groups to the browser Adding your own tracks to the browser Adding a new, custom (non-UCSC) genome to the browser Modifying the source code Custom Track Database Debugging the CGI binaries Notes on security Proxy support Support for cloud URLs The UDC local cache directory Activating CRAM support for the Genome Browser. Using FreeType font support for anti-aliased text Building the kent source tree. Adding a track hub to your hubPublic table so it appears under My Data > Track Hubs Overview of the Genome Browser directories and databases The genome browser requires only Apache and MariaDB and uses these directories: - static html files: we typically keep them under /usr/local/apache/htdocs and configure Apache to load them from there, to avoid conflicts with the distribution of the Linux default location /var/www/html - MariaDB databases: most of them are read-only, except the hgcentral database which is read-write. Most linux distributions keep these under /var/lib/mysql. (It is possible to get the genome browser to work with MySQL after version 8, but we do highly discourage it, as our download procedures use MyISAM .frm files which MySQL 8 dropped.) - static genome data files in /gbdb/ - binary CGI programs that generate images from the MariaDB and /gbdb files and write them into the trash directory (see below). We modify our Apache config to load CGIs from /usr/local/apache/cgi-bin, so as to not conflict with the default directory of the Linux distribution - a directory for temp files called trash, located in the parent directory of the CGI programs, usually /usr/local/apache/trash - a small text file hg.conf in the same directory as the CGI programs, with information on how to connect to MariaDB, the location of the other directories and various other settings, on our machines the location of this file is /usr/local/apache/cgi-bin/hg.conf - uploaded custom data gets written by the CGI programs into MariaDB databases in the database customtrash and also into files under /usr/local/apache/trash which is symlinked from /usr/local/apache/htdocs/trash so these files are accessible to Apache. When a web browser requests a Genome Browser page, typically /cgi-bin/hgTracks, Apache executes this CGI program. The programs then read information about how to connect to MariaDB using the file hg.conf, connects to MariaDB, reads the installed genome assemblies and the current user session from the MariaDB database hgcentral. For each genome assembly, there is a separate MariaDB database (e.g. hg38). Some types of data (e.g. raw genome sequences) are kept as indexed binary files outside of MariaDB, they are located in /gbdb, e.g. /gbdb/hg38. The location of the /gbdb directory can be changed with a setting in hg.conf. Some types of data are not specific for a genome, these are kept in the MariaDB databases hgFixed, proteome and visiGene. We strongly recommend to follow the default locations, and to place our CGI programs in /usr/local/apache/cgi-bin. The htdocs root directory for html files should then be in /usr/local/apache/htdocs. All Genome Browser components called from Apache get their settings from the central configuration file /usr/local/apache/cgi-bin/hg.conf. Among others, the location and the username/password for the MariaDB server is specified there. To load data into the genome browser databases, you need a command line tool like hgLoadBed. These tools are distributed separately from the CGI programs. Some tools create only MariaDB tables, others write into a /gbdb subdirectory. Most of them require a configuration file ~/.hg.conf in your home directory with the MariaDB connection information, like server name, username and password. The data loading is done from the Unix command line and not dependent on the CGI programs that create the Genome Browser graphics. Software Requirements To run our provided binaries: - Linux/Ubuntu/CentOS/Unix/MacOSX operating system - Apache2.x - http web server - http://httpd.apache.org/ - MariaDB development system and libraries - http://mariadb.com/ (MySQL 8 removed support for MyISAM schema files, which makes downloading our data file very cumbersome and slow) - libpng runtime and development packages - http://www.libpng.org/ - libssl runtime and development packages - http://www.openssl.org/ - Universally Unique Identifier library - http://e2fsprogs.sourceforge.net/ If you want to make modifications to our software, you need to compile it: - gnu gcc - C code development system - http://www.gnu.org/software/gcc/ - gnu make - http://www.gnu.org/software/make/ Optional: - 'ghostscript' ps to pdf converter - http://ghostscript.com - 'git' source code management: http://git-scm.com/downloads - 'gmt' map plotting tools http://www.soest.hawaii.edu/gmt/ - 'pstack' for stack traces - 'R' for the GTex track - 'python-mysqldb' for the gene interactions track (python2) It is best to install these packages with your standard operating system package management tools: - Debian/Ubuntu: apt-get install ghostscript apache2 mariadb-server gmt r-base uuid-dev python-mysqldb - Redhat/Fedora/CentOS: yum install libpng12 httpd ghostscript GMT hdf5 R libuuid-devel MySQL-python On newer distributions, python-mysqldb / MySQL-python is not available anymore. In this case, install python2, pip for it and then use pip to install the mysql library ("pip2 install MySQL-python"). See the file installer/browserSetup.sh for the commands. Hardware and disk space requirements We currently use the following hardware to support our website: - 24 CPUs and 128Gb of memory for each of the six machines - 16 CPUs, 64 Gb of memory for the MariaDB server The UCSC Genome Browser website experiences over one million hits per day. Your hardware requirements may be much less demanding and will depend upon how much traffic you expect for your mirror. Annotation database size differs a lot between the assemblies: The full size of the hg19 database in 2016 is 6 TB, for ce2 it is 5GB. It also depends on the tracks: The size of the hg19 annotations can be reduced to 2TB if you do not download any ENCODE tracks. The size of only the main gene and SNP annotations is around 5GB for hg19 and hg38. You can use the following command to get the size of the files for all of the assemblies, but it can also be modified to give the size for a particular assembly: rsync -hna --stats rsync://hgdownload.soe.ucsc.edu/gbdb/ | egrep "Number of files:|total size is" For example, to get the size of all of the files for hg19, you would use the following command: rsync -hna --stats rsync://hgdownload.soe.ucsc.edu/gbdb/hg19/ | egrep "Number of files:|total size is" After running that command, you should see output like this: Number of files: 54886 total size is 6515.70G speedup is 5181080.38 (DRY RUN) The next command will give you the size of the entire mySQL/MariaDB database, but can be changed to get the size for a particular assembly: rsync -hna --stats rsync://hgdownload.soe.ucsc.edu/mysql/ | egrep "Number of files:|total size is" Installing the UCSC Genome browser Note: we offer Genome-Browser-in-a-Box (GBIB), a fully configured virtual machine image that can be converted for VirtualBox, VMWare, Hyper-V and other popular environments. We also offer Genome-Browser-in-the-Cloud (GBIC) an shell script that installs a genome browser in most main Linux distributions (Most Debian and Redhat-based ones, like Ubuntu and CentOS). See https://genome.ucsc.edu/goldenPath/help/mirror.html Scripts to perform all of the functions below can be found in the directory https://github.com/ucscGenomeBrowser/kent/tree/master/src/product/scripts. In a git clone of the kent repository, the scripts are located in src/product/scripts. Confirm the following: 1. Apache web server is installed and working, http://localhost/ provides the Apache default home page from your machine NOTE: The browser static html web pages require the Apache XBitHack option to be enabled to allow SSI html statements to function. Add 'Options +Includes' for your html directory, your httpd.conf file entry looks like: XBitHack on <Directory /usr/local/apache/htdocs> Options +Includes </Directory> You can test your Apache cgi-bin/ directory by copying the script src/product/scripts/printEnv.pl into it. 2. MariaDB database is installed and working mysql -u browser -pgenome -e 'show tables;' mysql MariaDB can be run from the command line, and the tables from the database MariaDB can be displayed. MariaDB development package is installed (mariadb-devel on RedHat) The directory: /usr/include/mysql/ has the mysql .h files And the library: /usr/lib/mysql/libmysqlclient.a exists (your exact pathnames may vary depending upon your installation) Set MySQL/MariaDB database access permissions. The examples mentioned in the "Mysql setup" section will allow this setup to function as described here. To setup the example user accounts as mentioned in these instructions, run the script: ex.MySQLUserPerms.sh 3. Find the location of your Apache WEB server DocumentRoot and cgi-bin directory. Typical locations are: /var/www and /usr/local/apache, /var/www/html, /var/www/cgi-bin The directory where these are located is referred to as WEBROOT in this documentation: WEBROOT=/var/www export WEBROOT The browser WEB pages and cgi-bin binaries expect these two directories to be next to each other in ${WEBROOT} since referrals in html are often: "../cgi-bin" The browser should function even if WEBROOT is in a different directory from the primary Apache web root. In this case, the three directories: html cgi-bin and trash should be at the same level in this other WEBROOT. For example: /some/other/directory/path/html/ /some/other/directory/path/cgi-bin/ /some/other/directory/path/trash/ Symlinks to the trash directory should exist from the html directory. As so: /some/other/directory/path/html/trash -> ../trash The actual trash directory can be somewhere else. If it is not in your Apache /var/www/trash/ directory, then create that symlink as well as the html/trash symlink. For example: /var/www/trash -> /some/other/directory/trash /var/www/html/trash -> /some/other/directory/trash 4. Create html, cgi-bin and trash directories: mkdir ${WEBROOT}/html mkdir ${WEBROOT}/cgi-bin chmod 755 ${WEBROOT}/cgi-bin (this chmod 755 will prevent suexec failures that are indicated by "Premature end of script headers" errors in the Apache error_log. Your cgi binaries should also be 755 permissions.) mkdir ${WEBROOT}/trash chmod 777 ${WEBROOT}/trash ln -s ${WEBROOT}/trash ${WEBROOT}/html/trash The browser creates .png (and other) files in the trash directory. The 'chmod 777' allows the Apache WEB server to write into that directory. A cron job should be set to periodically clean the files in trash. See also, the two scripts here: src/product/scripts/trashCleanMonitor.csh src/product/scripts/trashCleaner.csh 5. Download static WEB page content: See also: src/product/scripts/updateHtml.sh 6. Copy CGI binaries: This set of binaries are for x86_64 types of Linux machines. If you need to instead build binaries for your platform, follow the instructions in the section "Building the kent source tree", below. See also: src/product/scripts/kentSrcUpdate.sh rsync -avP rsync://hgdownload.soe.ucsc.edu/cgi-bin/ ${WEBROOT}/cgi-bin/ 7. Create hgcentral database and tables. This is the primary gateway database that allows the browser to find specific organism databases. See also: scripts/fetchHgCentral.sh to fetch a current copy of hgcentral.sql mysql -u browser -pgenome -e "create database hgcentral;" mysql -u browser -pgenome hgcentral < hgcentral.sql Please note, it is possible to create alternative hgcentral databases. For example, for test purposes. In this case use a unique name for the hgcentral database, such as "hgcentraltest", and it can be specified in the hg.conf file as mentioned in the next step. To create a second copy of the hgcentral database: mysql -u browser -pgenome -e "create database hgcentraltest;" mysql -u browser -pgenome hgcentraltest < hgcentral.sql 8. Create the hg.conf file in ${WEBROOT}/cgi-bin/hg.conf to allow the CGI binaries to find the hgcentral database Use the file here: ex.hg.conf as the beginning template for your system: Copy the sample hg.conf: cp ex.hg.conf ${WEBROOT}/cgi-bin/hg.conf Please edit this hg.conf file and set any parameters required for your installation. Use the comments in that file as your guide. Browser developers will want a copy of this file in their home directory with mode 600 and named: ~/.hg.conf These copies may have different db.user specification to allow developers write access to the database. 9. Load databases of interest. See also: src/product/scripts/activeDbList.sh src/product/scripts/minimal.db.list.txt src/product/scripts/loadDb.sh And discussion in scripts/README about whether you can use directly the MariaDB binary database files, or if you need to download goldenPath database text dumps and load them into the database. If you use MariaDB, you can use the binary files, with MySQL >= 8 you need to use dumps, which is why we discourage the use of MySQL >= 8 with the Genome Browser. An alternative to loading the database tables from text files, is to directly rsync the MariaDB tables themselves and place them in your MariaDB /var/ directory. These tables are much larger than the text files due to the sizes of indexes created during a table load, but it can save a lot of time since the data loading step is quite compute intensive. A typical rsync command for an entire database (e.g. ce4) would be something like: rsync -avP --delete --max-delete=20 rsync://hgdownload.soe.ucsc.edu/mysql/ce4/ /var/lib/mysql/ce4/ 10. Download extra databases to work with a full genome assembly such as human/hg38: hgFixed go140213 proteins140122 sp140122 Construct symlinks in your MariaDB data directory to use database names: go proteome uniProt for these database directories: $ ls -og proteome go uniProt lrwxrwxrwx 1 8 Feb 26 11:39 go -> go140213 lrwxrwxrwx 1 14 Mar 27 12:01 proteome -> proteins140122 lrwxrwxrwx 1 8 Mar 27 12:01 uniProt -> sp140122 $ ls -ld go140213 proteins140122 sp140122 drwx------ 2 mysql mysql 4096 Feb 26 10:57 go140213 drwx------ 2 mysql mysql 4096 Aug 19 08:08 proteins140122 drwx------ 2 mysql mysql 4096 Mar 26 13:01 sp140122 These file names are data stamped YYMMDD to indicate changes over time as they are updated with new builds of the UCSC gene track. When a new UCSC gene track is released, fetch new databases and change the symlink. 11. Copy the gbdb data to /gbdb - See also: scripts/fetchFullGbdb.sh scripts/fetchMinimalGbdb.sh 12. The browser should now appear at the URL: http://localhost/ Check your Apache error_log file for hints to solving problems. 13. BLAT server setup: The blatServers table in the database hgcentral needs to have a fully qualified host name specified in the 'host' column. Educational and non-profit institutions are allowed to use blat free of charge. Commercial installations of the browser require a license for blat. See also: http://kentinformatics.com/index.html and: http://genome.ucsc.edu/license/ In the source tree: src/gfServer/README.blat 14. Useful links: http://genomewiki.ucsc.edu/index.php/Category:Mirror_Site_FAQ There are numerous README files in the source tree on a variety of specific subjects, e.g.: ./src/README ./src/product/README.* ./src/hg/makeDb/trackDB/README ./src/hg/makeDb/doc/make*.txt 15. Apache configuration: To lock down your trash directory from scanning via "indexes" enter the following in your httpd.conf: <Directory "/var/www/html/trash"> Options MultiViews AllowOverride None Order allow,deny Allow from all </Directory> The specified directory name is your apache: DocumentRoot/trash e.g. /usr/local/apache/htdocs/trash MariaDB Setup 1. Enable "LOAD DATA LOCAL INFILE": Set these in /etc/my.cnf or /etc/mysql/my.cnf: [mysqld] local-infile=1 [client] local-infile=1 2. MariaDB Storage Engine: In recent versions of MySQL/MariaDB, the default storage engine has changed from myisam to innodb. However the myisam engine should be used with the UCSC Genome Browser. Set it in /etc/my.cnf or /etc/mysql/my.cnf: [mysqld] default-storage-engine=MYISAM Always restart your MariaDB server after making changes to these configuration files. 3. Users: There are three cases of identity to consider when providing access to the MariaDB system for the browser CGI binaries and browser developers: 1. A MariaDB user that needs read-only access to the genome databases. The browser CGI binaries require read-only access to the genome databases. 2. A MariaDB user that has write permissions to one database. The CGI binaries require write permissions to one particular database (hgcentral) for maintaining user's cart information to store the user's browser cookie settings. 3. A MariaDB user that has general write permissions to all browser and genome databases to be used by developers The cgi-bin binaries obtain the first two of these MariaDB identities from the text file: $WEBROOT/cgi-bin/hg.conf Developers of the browser databases obtain their MariaDB identities from a text file in their home directory: ~/.hg.conf Note the initial dot in the name: .hg.conf This file in a user's directory will specify a higher-privileged user to allow read/write access to the MariaDB databases. This file must be set to mode 600 to provide security of the user and password to the database: $ chmod 600 ~/.hg.conf All kent source code commands use this file to access the MariaDB databases. Since this file contains password information it requires the permissions to be set at 600 to keep it secret. The kent source code commands will enforce this access and not function unless it is set at 600 permissions. Therefore you will want to create three different MariaDB users for these purposes. The examples listed below are implemented in the shell script: src/product/scripts/ex.MySQLUserPerms.sh You can execute that script to set up these example users. An example full read/write access user: "browser", is created with the following procedure. For the following it is assumed that your root account has access to the MariaDB database. You should be able to perform the following: $ export SQL_PASSWORD=mysql_root_password $ mysql -u root -p${SQL_PASSWORD} -e "show tables;" mysql Create a MariaDB user called "browser" with password "genome" and give access to selected MariaDB commands for the following list of databases. When you add other databases, you will need to add these permissions to your databases. This procedure of adding permissions specifically for a set list of databases is a more secure method than allowing the MariaDB "browser" user to have access to any database. ( MySQL version 5.5 requires the LOCK TABLES permission here ) ( FILE, CREATE, DROP, ALTER, LOCK TABLES, CREATE TEMPORARY TABLES on ${DB}.* ) for DB in cb1 hgcentral hgFixed hg38 proteins140122 sp140122 go140213 uniProt go proteome do mysql -u root -p${SQL_PASSWORD} -e "GRANT SELECT, INSERT, UPDATE, DELETE, \ FILE, CREATE, DROP, ALTER, CREATE TEMPORARY TABLES on ${DB}.* \ TO browser@localhost \ IDENTIFIED BY 'genome';" mysql done The above granted permissions are recommended for browser developers. The WEB browser CGI binaries need SELECT, INSERT and CREATE TEMPORARY TABLES permissions. For example, you should create a special user for the browser genome databases only. In this example, user: "readonly" with password: "access" for DB in cb1 hgcentral hgFixed hg38 proteins140122 sp140122 go140213 uniProt go proteome do mysql -u root -p${SQL_PASSWORD} -e "GRANT SELECT \ on ${DB}.* TO \ readonly@localhost IDENTIFIED BY 'access';" mysql done Create a database to hold temporary tables: mysql -u root -p${SQL_PASSWORD} -e "create database hgTemp" mysql -u root -p${SQL_PASSWORD} -e "GRANT SELECT, INSERT, \ CREATE TEMPORARY TABLES \ on hgTemp.* TO \ readonly@localhost IDENTIFIED BY 'access';" mysql A third MariaDB user should be created with read-write access to only the hgcentral database. For example, a user: "readwrite" with password: "update" for DB in hgcentral do mysql -u root -p${SQL_PASSWORD} -e "GRANT SELECT, INSERT, UPDATE, DELETE, \ CREATE, DROP, ALTER on ${DB}.* TO readwrite@localhost \ IDENTIFIED BY 'update';" mysql done The cgi-bin binaries obtain their MariaDB identities from the hg.conf file in the cgi-bin directory. The file in this directory: src/product/ex.hg.conf demonstrates the use of the "readonly" user for genome database access and the "readwrite" user for hgcentral database access. 4. The hgsql command: Developers can access the browser databases via the 'hgsql' command which can be built in the source-tree at: kent/src/hg/hgsql/ This 'hgsql' command provides a convenient front-end to the standard 'mysql' command by reading the user's ~/.hg.conf file to provide access to the browser databases with the appropriate identity. Each user creates a ~/.hg.conf file (same format as the above mentioned cgi-bin/hg.conf file) and the specified database user identity is used for accesses to the browser databases. This same function of reading ~/.hg.conf for database access is built into all the source-tree binaries which modify the genome databases. The above example hg.conf could be used as a user's ~/.hg.conf file with the change of db.user, db.password, central.user, and central.password to be the fully permitted read-write user: db.user=browser db.password=genome central.user=browser central.password=genome central.db=hgcentral To test this access with your ~/.hg.conf file in place: hgsql -e "show tables;" hgcentral hgsql -e "show grants;" hgcentral 5. Configuring MariaDB SSL connections (entirely optional, only needed if your IT department requires it): MariaDB is typically compiled with SSL capability from OpenSSL or yaSSL. To see if your server supports ssl, login to MariaDB and run this command: mysql> show variables like '%ssl%'; +---------------+----------+ | Variable_name | Value | +---------------+----------+ | have_openssl | DISABLED | | have_ssl | DISABLED | | ssl_ca | | | ssl_capath | | | ssl_cert | | | ssl_cipher | | | ssl_crl | | | ssl_crlpath | | | ssl_key | | +---------------+----------+ If your MariaDB was compiled with SSL support, which is true of virtually all MariaDB packages being provided today, you can easily enable SSL by adding settings to /etc/my.cnf: ------- my.cnf: ------- [mysqld] ssl ssl-key=/somepath/server-key.pem ssl-cert=/somepath/server-cert.pem ssl-ca=/somepath/ca.pem ssl-capath=/somepath/ ssl-cipher=DHE-RSA-AES256-SHA:AES128-SHA # mysql 5.6.3 or later ssl-crl=/someCrlPath/some-crl.pem ssl-crlpath=/someCrlPath/ # mysql5.7 or later require all connections to be encrypted require_secure_transport server After making changes to my.cnf, be sure to restart your mariadb service. The key means private key here, and should be kept secured. The cert is a certificate acting like a public key, signed by a trusted authority (CA). If a key and cert are available, that means you can authorize. And it proves the key exists. The key is not sent to the other party. The cert is. If a ca is available it can show what certs to trust. You do not need all the settings, but some versions of MariaDB do not activate SSL unless at least one of these is found: key, cert, ca, capath, cipher If you configure a key for the server or client, you will also provide its cert. We cannot teach you how to create SSL certificates here. There are many websites including MariaDB that have information about making keys and certs and ca. If you just add the ssl option to the top, it will try to use SSL, or make it available. The ca is the certificate authority cert that you are using. It could be just a local self-signed authority you made up, or it can be a commercial authority like veriSign. This typically is used to sign the certificate for the server and users. The capath is a directory where ca-certs exist (OpenSSL only). The crl is a certificate revocation list. (OpenSSL only). The crlpath is a directory where revocation lists exist (OpenSSL only). This crl options are a new feature in 5.6.3, not sure it works right yet. After making a key for the server, and signing a cert for it with ca, you can create SSL connections. Do not specify a passphrase when creating your server keys. The cipher setting is a colon-separated list of SSL ciphers that are supported. The security files like certs etc. that are specified in the above settings must be readable by the unix account that mysqld runs under, default is "mysql". SELinux or apparmor may block access to certain locations. /etc/mysql is the default location for .pem files on some platforms. yaSSL, which is still often used with the MySQL Community Edition, expects keys to be in the PKCS #1 format and doesn't support the PKCS #8 format used by OpenSSL 1.0 and newer. You can convert the key to the old format using openssl rsa: openssl rsa -in key_in_pkcs1_or_pkcs8.pem -out key_in_pkcs1.pem yaSSL requires that all components of the CA certificate tree be contained within a single CA certificate file and that each certificate in the file has a unique SubjectName value. To work around this limitation, concatenate the individual certificate files comprising the certificate tree into a new file and specify that file as the value of the --ssl-ca option. For example, cd my-certs-dir cat ca-cert.pem server-cert.pem (etc) > yaSSL-ca-cert.pem chmod +r yaSSL-ca-cert.pem Now use my-certs-dir/yaSSL-ca-cert.pem for certificate authority (ca) for clients. These are the SSL settings which can be placed into your hg.conf for CGIs or .hg.conf for utility programs: db.key=/sompath/someuser-key.pem db.cert=/sompath/someuser-cert.pem db.ca=/somepath/ca.pem db.caPath=/somepath db.crl=/someCrlPath/some-crl.pem db.crlPath=/someCrlPath/ db.verifyServerCert=1 db.cipher=DHE-RSA-AES256-SHA:AES128-SHA The key and certificate for "someuser" above are signed by a ca. The verifyServerCert setting if it exists tells the client to verify that the CN field in the server's cert matches the hostname to which it is connecting. This prevents Man-In-the-Middle attacks. The caPath and crlPath options only work with OpenSSL. The example shows the most common use for the profile "db". But the SSL settings work with any profile in the hg.conf file. Of course you can stick SSL settings into your [client] section of my.cnf, but the CGIs and utils would not see them. Only mysql and hgsql would see them. Configuring SSL requirements for MariaDB user accounts: You can tell MariaDB to require SSL for a user's account like this: GRANT ALL PRIVILEGES ON *.* TO 'someuser'@'%' REQUIRE SSL; You can tell MariaDB to use SSL for a user's account and to further require the client to use their key and x509 certificate to connect by saying: GRANT ALL PRIVILEGES ON *.* TO 'someuser'@'%' REQUIRE x509; There are more-specific requirements that may be added: GRANT ALL PRIVILEGES ON *.* TO 'someuser'@'%' REQUIRE SUBJECT '/C=US/ST=CA/L=Santa Cruz/O=YourCompany/OU=YourDivision/CN=someuser/emailAddress=someuser@YourCompany.com' AND ISSUER '/C=US/ST=CA/L=Santa Cruz/O=YourCompany/OU=YourDivision/CN=YourCompanyCA/emailAddress=admin@YourCompany.com' AND CIPHER 'DHE-RSA-AES256-SHA'; You can see the cert details like this: openssl x509 -in /somepath/someuser-cert.pem -text In later versions of MariaDB, it is a requirement that the CN of the CA cert must DIFFER from the CN of the user and server certs. Further MySQL SSL documentation is available from https://dev.mysql.com/doc/refman/5.6/en/creating-ssl-files-using-openssl.html Local Git repository (aka: "the source tree") Use the following procedures to create your own personal copy of the kent source tree where you can have your own edits that are not part of the development at UCSC This is useful for mirror sites that have their own customizations in the source tree for local circumstances. It will also be necessary if you want to add your own tracks to your mirror (see next section). Install Git software version 1.6.2.2 or later. See the Git Community Handbook installation (https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) and setup (https://git-scm.com/book/en/v2/Getting-Started-First-Time-Git-Setup) instructions, as well as our Installing Git (http://genomewiki.ucsc.edu/index.php/Installing_git) Genomewiki page. Start an initial Git local repository: git clone git://genome-source.soe.ucsc.edu/kent.git or, if a firewall prevents git daemon port 9418, use: git clone http://genome-source.soe.ucsc.edu/kent.git The kent source tree will be imported to the current working directory in a directory named ./kent/. Track the beta branch at UCSC repository: Most users want to use the beta branch, which has tested, released versions of the browser. To create a beta tracking branch: cd kent git checkout -t -b beta origin/beta The -b creates a local branch with name "beta", and checks it out. The -t makes it a tracking branch, so that 'git pull' brings in updates from origin/beta, the "real" beta branch in our public central read-only repository. To get the latest UCSC release, from anywhere within the kent source tree: git pull Updates: UCSC generally updates the origin/beta branch every three weeks. If you are updating database tables for a mirror site, we recommend that you update the source at the same time, as source code is sometimes modified to include operations on new columns that have been added to database tables. For instructions on keeping local tracks separate from UCSC Genome Browser tracks created at UCSC and mirrored from there, see the section "Adding tracks to the browser" below. Adding your own track groups to the browser If you want to add your own tracks (see next section), you probably want to put them into a separate track group, so they are visually separated from the tracks provided by UCSC. The MariaDB table grp contains the list of all track groups. If you rsync the data from UCSC on a regular schedule, the table would be overwritten each time. To avoid this, you can create an empty table with the same schema, e.g. in the database hg38: CREATE TABLE grp_local LIKE grp; You can then use the MariaDB INSERT statement to add a new track group to this table, specify the name, label, priority and whether the group should be closed by default (most are open by default). INSERT INTO grp_local VALUES ('test', 'This is my group', 1, 0); Then, edit cgi-bin/hg.conf and add a line like this: db.grp=grp_local,grp This means that grp_local is added to the contents of grp and grp_local has higher priority, so you can override the UCSC-provided default groups, if needed. This will not have any effect yet. First you need to add a new track that uses your new group. You can use your new group's name using the "group" statement in trackDb (see the next section). All tracks with a group not in the grp table will end up in the group "Experimental" at the bottom of the page. Adding your own tracks to the browser A track needs two items to make it exist in the browser: 1. A database table with the track data 2. An entry in a database table: trackDb_localTracks Built from track specifications in your trackDb.ra file. The format of the trackDb.ra file is explained at http://genome.ucsc.edu/goldenPath/help/trackDb/trackDbDoc.html The correspondence between the database table and the trackDb.ra definition is in the name used on the 'track' line in the trackDb.ra file. Your database table name is defined by the 'track' definition line. To direct the genome browser to this trackDb_localTracks table to use as extra trackDb definitions, add this line to your cgi-bin/hg.conf file: db.trackDb=trackDb_localTracks,trackDb The order matters. Any definitions for tracks in trackDb_localTracks will override any definitions for the same named tracks in trackDb. You can then override the standard definitions for UCSC-defined tracks. The usual case will be that your tracks are unique to your local installation. Almost all of the database tables have specific loader programs to load the track data. The loader programs also verify the data before it is added to the table, and they create the proper indexes on the table to allow efficient display by the genome browser. By far the most common format of track data is the BED format. See also: http://genome.ucsc.edu/FAQ/FAQformat.html#format1 for a description of BED file formats. A typical BED file format is loaded into a database table with the loader: hgLoadBed For example, to load the data from the file: data.bed into the table named: bedExample hgLoadBed hg17 bedExample data.bed You then add a section that starts with the line "track bedExample" to your trackDb.ra file, run hgTrackDb to create the trackDb_localTracks database table and the table should appear, as long as trackDb_localTracks has been added to hg.conf as explained before. There are a variety of file formats: GFF, GTF, PSL, WIG, MAF as well as a variety of specialized data types. All the loader programs can be seen in the source tree as subdirectories in: src/hg/makeDb/ cd src/hg/makeDb ls -d hg* The build instructions for the browser code do not include instructions for building all of the loaders, or other utilities in the kent source tree. This is because there are literally hundreds of utilities, 345 at last count, that are not needed for ordinary browser development. In most cases a developer will need only a couple of the loaders and utilities. Since the libraries were built for the CGI binaries, to build any utility or loader, simply go into its directory and run a 'make'. If you do not have the kent tree source repository cloned with git yet onto your own disk, please go back to the previous section and do that now. For our purposes here, we need for example, for BED format tracks: 1. hgLoadBed 2. hgTrackDb 3. hgFindSpec To build the three loaders mentioned, go to the three directories in the kent git source repository: src/hg/makeDb/hgTrackDb/ src/hg/makeDb/hgFindSpec/ src/hg/makeDb/hgLoadBed/ And run a 'make' in each one. The resulting binary is placed in: \(HOME/bin/\)MACHTYPE This binary directory should be in your PATH, or make this directory be a symlink to some binary directory that is in your PATH and you have write permission to. See also: new assistant scripts as of March 2010 in the src/product/scripts/ directory here to fetch and build the source tree. If you want to build all the utilities and all database loaders now, perform the following 'make' commands in your source tree: cd src make clean make libs cd hg make cd ../utils make This builds everything cleanly, all CGI binaries, all database loaders, all utilities. Perform this sequence each time you do a 'git pull' on your source tree. The 'make clean' step is especially important since the makefile hierarchy does not have built in dependencies and will not rebuild items that depend upon each other. The traditional dependency on the source tree libraries is taken care of because a make in any directory that produces a binary will always re-link the binary every time, thus always picking up any potentially new library. With those three loader programs built, you can now load BED format tracks, and build the trackDb_localTracks table as mentioned next. The hgTrackDb and hgFindSpec loaders are used to build the trackDb and hgFindSpec tables in the database. You can obtain example trackDb entries from the source tree hierarchy: src/hg/makeDb/trackDb/ in any of the *.ra files. And you will need to refer to the README file in that directory for information about options you can use with each track type or use our full trackDb.ra documentation at http://genome.ucsc.edu/goldenPath/help/trackDb/trackDbDoc.html. To work independently of the UCSC source tree, establish your own trackDb.ra files outside the UCSC source tree in a directory of your choice under your control. Then, to load them into the database, run the hgTrackDb command with this simple makefile in the directory where your .ra file exists: trackDbSql=/path/to/kent/source/tree/src/hg/lib/trackDb.sql DB=hg19 all:: hgTrackDb . ${DB} trackDb_localTracks ${trackDbSql} . This hgTrackDb command reads your trackDb.ra file and converts it into row entries for each track specified in it into row contents in this new table trackDb_localTracks. The DB= specification is your database of interest, this example: hg19 This loads your local specific table trackDb_localTracks in the database. This name trackDb_localTracks is not special, just different than the ordinary trackDb table. It should have some meaning to anyone in your environment and not be the same name as any UCSC database table. The two '.' arguments in the command above refer to directory names. Since you have no hierarchy of levels in this single directory, unlike in the source tree trackDb hierarchy, the '.' arguments refer to the current directory. See also: - A similar overview: http://genomewiki.ucsc.edu/index.php/Local_tracks_at_mirror_sites - TrackDb.ra track configuration format: http://genome.ucsc.edu/goldenPath/help/trackDb/trackDbDoc.html Adding a new, custom (non-UCSC) genome to the browser Please note that setting up an assembly hub is a lot easier than adding a genome to a local mirror. The browser can be made to operate with a bare minimum of tables for the purpose of demonstrating the CGI binaries are functioning. The only tables you need to load for this are: 1. all tables in the hgcentral database 2. six tables in the human genome Create an empty hgcentral database: $ hgsql -e "create database hgcentral;" mysql Load all tables into the hgcentral database. Copy all the mysql data files from rsync -avP rsync://hgdownload.soe.ucsc.edu/mysql/hgcentral/ . directly into the MySQL data area for your hgcentral database. (something usually like /var/lib/mysql/hgcentral/) Or load this database with mysql/hgsql commands and the hgcentral.sql text file dump of these tables from: rsync -avP rsync://hgdownload.soe.ucsc.edu/genome/admin/hgcentral.sql . And then six tables for the latest human database. The gateway page always needs a minimum human database in order to function even if the browser is being built for the primary purpose of displaying other genomes. This default can currently be changed in the source tree in src/hg/lib/hdb.c (to be done: specify this default in hg.conf file) Start with an empty database, for example hg18: hgsql -e "create database hg18;" mysql Again, copy the MariaDB files directly from the download server, for example hg18: rsync -avP rsync://hgdownload.soe.ucsc.edu/mysql/hg18/ . (beware, this is several TB of data) into your MariaDB data area. Or load these tables from the text SQL dumps from: rsync -avP rsync://hgdownload.soe.ucsc.edu/goldenPath/hg18/database/ . (beware, this is several TB of data) The minimal set of tables required are: grp trackDb hgFindSpec chromInfo gold gap With this set of six tables the gateway page will begin to function and the browser page and table browser will function. Other browser functions are not ready yet without additional tables and databases. This is a bare minimum just to demonstrate the CGI binaries are working. This will all work even without copying any files for the /gbdb/ data area, although most functions will not work, such as fetching the DNA sequence from a browser view. The DNA sequence for an assembly is found in, for example hg18: /gbdb/hg18/nib/chr*.nib Some assemblies have all the DNA sequence in a single .2bit file, for example: /gbdb/mm8/mm8.2bit Modifying the source code If you want to make changes to the source code, contact us first via the mailing list, to make sure that there is no option in development or an undocumented way to solve your problem. If you need to change the code, make sure to isolate your changes into a single function, if possible. Using git, merge your branch into our "beta" branch, ideally for every release, then recompile. If your changes could be useful for someone else, and you are getting tired of updating them to keep up with our changing code base, consider submitting them as a pull request, so we can integrate it into the main code base and you do not have to worry about updating them anymore. Once you have git setup properly, merging your changes into our current release should be as easy as this: git pull # get new version git checkout beta # switch to our stable branch git merge myChangesBranch # merge your changes into the beta branch make -j 20 cgi-alpha # compile and put CGIs into /usr/local/apache/cgi-bin Custom Track Database Without any specific hg.conf configuration, custom track data is kept in flat files in the /trash/ct/ directory. It is much more efficient to load them into a MariaDB database. This article discusses the steps required to enable this function. 1. Summary configuration - database loader binaries hgLoadBed, hgLoadWiggle and wigEncode are installed in /cgi-bin/loader/ - these are installed via the normal 'make cgi' in the source tree kent/src/hg/ directory or via rsync. They are probably aleady in your cgi-bin directory. - an empty customTrash database has been created on the MariaDB host - create this manually once, the MariaDB host name is a configuration item, the database name customTrash is not a configuration item - temporary read-write data directory /data/tmp has been created with read/write/delete enabled for the Apache server effective user, this directory name is a configuration item - configuration items are specified in /cgi-bin/hg.conf/ - this will turn on the function - for command line access to the database, create a special ~/.hg.ct.conf to be used with the environment variable HGDB_CONF - create a cron job to run a cleaner script to expire and remove older tables from the database - dbTrash command is used for this purpose 2. Host and database name For performance and security considerations, the MariaDB host for the custom track database can be a separate machine from the ordinary MariaDB host that usually serves up the assembly databases or the hgcentral database. It is not required that the custom track database be on a separate MariaDB server. The specification of the host machine is placed in the /cgi-bin/hg.conf file, for example a host machine called "ctdbhost": customTracks.host=ctdbHost The database name used on this host is fixed at customTrash which is a define in the source tree file hg/inc/customTrack.h Edit /cgi-bin/hg.conf configuration items: The following items must be specified in /cgi-bin/hg.conf to enable this function: customTracks.host=ctdbhost customTracks.user=ctdbuser customTracks.password=ctdbpasswd customTracks.useAll=yes Establish this user account and password in MariaDB with db and user privileges: Select, Insert, Update, Delete, Create, Drop, Alter, Index for example with your MariaDB root user account: hgsql -hctdbhost -uroot -p -e \ "GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,DROP,ALTER,INDEX" \ on customTrash.* TO ctdbuser@yourWebHost IDENTIFIED by 'ctdbpasswd';" mysql Optionally, a temporary read-write directory used during database loading can be specified: customTracks.tmpdir=/data/tmp The default for this is /data/tmp and should be created with read/write/delete access for the Apache server effective user. It should be on a local filesystem for best access speed, not via NFS. 3. Database loaders: The database loaders used to load custom tracks are the standard loader commands found in the source tree, hgLoadBed, hgLoadWiggle and wigEncode. They are installed into /cgi-bin/loader/ with a 'make cgi' from the source tree directory kent/src/hg/ These loaders are used by the cgi binaries hgCustom, hgTracks, and hgTables to load custom tracks into the database. They are operated in an exec'd pipeline fashion, the code details can be see in src/hg/lib/customFactory.c 4. Command line access: Since the MariaDB host may be different than your ordinary MariaDB host, you will need to create a unique $HOME/.ct.hg.conf file to be used in the case where you want to manipulate this separate database with the kent source tree command line tools. This unique .ct.hg.conf is merely a copy of your normal .hg.conf file but with a different host/username/password specified: db.host=ctdbhost db.user=ctdbuser db.password=ctdbpasswd central.db=hgcentral Remember to set the privileges on this hg.conf file at 600: chmod 600 $HOME/.ct.hg.conf To enable the use of this file for subsequent command line operations, set the environment variable HGDB_CONF to point to this file, for example in the bash shell: export HGDB_CONF=$HOME/.ct.hg.conf With that in place, you can examine the contents of the customTrash database: hgsql -e "show tables;" customTrash This unique hg.conf file will also be used by the cleaner command dbTrash 5. Cleaner script The database and the temporary data directory /data/tmp need to be kept clean. This is similar to the current cleaner script you have running on your /trash filesystem. In this case there is a specific source tree utility used to access and clean the database. The temporary data directory /data/tmp would stay clean if each and every loaded custom track was successfully loaded. In the case of badly formatted or illegal data submitted for the custom track, the database loaders do not remove their temporary files from /data/tmp This /data/tmp directory can be kept clean with, for example, an hourly cron job that performs: find /data/tmp -type f -amin +10 -exec rm -f {} \; This would remove any file not accessed in the past 10 minutes. The database cleaner command dbTrash should be run as a cron job encapsulated in a shell script something like this, which maintains a record of items cleaned to enable later analysis of custom track database usage statistics: #!/bin/sh DS=`date "+%Y-%m-%d"` YYYY=`date "+%Y"` MM=`date "+%m"` export DS YYYY MM mkdir -p /data/trashLog/ctdbhost/${YYYY}/${MM} RESULT="/data/trashLog/ctdbhost/${YYYY}/${MM}/${DS}.txt" export RESULT /cluster/bin/x86_64/dbTrash -age=48 -drop -verbose=2 > ${RESULT} 2>&1 Running this once a day will remove any tables not accessed within the past 48 hours. The dbTrash command is found in the source tree in kent/src/hg/dbTrash The /trash directory can be kept clean with the following two commands, one to implement an 8 hour expiration time on most files, the second to implement a 48 hour expiration time on custom track files: find /trash \! \( -regex "/trash/ct/.*" -or -regex "/trash/hgSs/.*" \) \ -type f -amin +480 -exec rm -f {} \; find /trash \( -regex "/trash/ct/.*" -or -regex "/trash/hgSs/.*" \) \ -type f -amin +2880 -exec rm -f {} \; 6. metaInfo and history You will note two special and persistent tables in the customTrash database: metaInfo and history. The metaInfo table records a time of last use for each custom track table and a useCount for statistics. The time of last use is used by the cleaner utility dbTrash to expire older tables. The history table is the same as the history table in the normal assembly databases. The loader commands, hgLoadBed and hgLoadWiggle record into the history table each time they load a track. The cleaner command dbTrash also records in the history table statistics about what it is removing. 7. Turning On Considerations Please note, if there are currently existing custom tracks in /trash/ct/ files, at the time of adding the configuration items to /cgi-bin/hg.conf/ those existing tracks will be converted to database versions upon their next use by the user. Therefore, to enable this function on the round-robin WEB servers, we will need to do the update to /cgi-bin/hg.conf in as much a simultaneous manner as possible. Perhaps something like a shell script to do eight background rsync's all at the same time. 8. Use of trash files with the database on When the custom tracks database is in use, there are still small files kept in /trash/ct which become the reference pointers to the actual database tables belonging to that custom track. The standard trash cleaner script should still be kept running to clean these files. 9. Known difficulties For the case of a custom track submission that contains more than one track set of data, in the case where one of the sets of data is illegal and causes a loading problem, even though some sets of data may have loaded successfully, the submitting user will see an error about the corrupted data, and they would need to correct their data submission to get all tracks successfully loaded. It remains to be seen just how good the error reporting system is for illegal data. Debugging the CGI binaries The typical sign of trouble is an Error 500 display in your web browser when accessing the CGI binaries, and the following message in your Apache error log: [Fri Mar 25 11:02:40 2005] [error] Premature end of script headers: hgTracks This is usually a simple configuration problem. Items to verify: 1. the hg.conf file in the cgi-bin directory specifies the correct user names and passwords for MariaDB database access. See also the section "MariaDB Setup" below. 2. The cgi-bin directory is set to permissions 755 and not 775 or 777 When permissions are too permissive for this directory, Apache errors out with suexec permission violations. 3. Verify change history of the database hgcentral. Rarely, changes in this database require corresponding changes in the source code. Make sure your code and version of hgcentral are synchronized. Newer versions of hgcentral database with old source code are OK. The problem is when you have new source code that expects new features in hgcentral. If these items are OK, then you can check the actual operation of a cgi binary. Go to the source tree directory of the cgi binary, for example hgTracks: kent/src/hg/hgTracks In this directory, run a 'make compile' to produce a binary that is left in this directory. This binary can be run from the command line: ./hgTracks By itself with no arguments, it should produce the default tracks display HTML page for the Human genome. This assumes you have set up your $HOME/.hg.conf file to allow access to the MariaDB databases. (See also: section "MariaDB Setup"). A binary execution failure should be obvious at this stage of the game. If it exits because of SIGSEGV we can run it under a debugger for specifics. More on this below. If the problem is specific to a particular set of tracks being displayed, or particular genomes or options, command line arguments can be given to these CGI binaries to provide the URL inputs that a CGI binary would normally see. To prepare the binaries for operation under a debugger, go to the src/inc directory and edit the common.mk configuration file. Change "COPT=-O" to read: "COPT=-g" GNU gcc will allow "-O" with "-g", and some bugs will only exhibit themselves with -O on. However the optimizations with -O can sometimes confuse the debugger's sense of location due to optimization rearrangement of code. Also eliminate the -Wuninitialized option from the HG_WARN definition to avoid constant warnings about that being incompatible with -g. Rebuild the source tree: cd kent/src make clean make libs cd hg/hgTracks make compile The hgTracks binary will now have all symbol information in it and it can be operated under a debugger such as ddd (or gdb, etc...). For the case of specific options or tracks causing problems, to find the full set of options in effect for the failure case, when your WEB browser is at the Error 500 display page, edit the displayed URL in your WEB browser to call the cgi binary cartDump: http://<your server>/cgi-bin/cartDump This will display all environment variables in effect at the time of the crash. Most of the track display options that are marked as "hide" can be ignored. That is their default setting already. The important ones are the db, position, and specific options for the track under consideration. The command line can be formatted just as if it was a URL string. For example: ./hgTracks "db=hg17&trackControlsOnMain=0&position=chr4:56214201-56291736" Or with spaces between the arguments: ./hgTracks db=hg17 trackControlsOnMain=0 position=chr4:56214201-56291736 Remember to protect special characters on the command line from shell interpretation by appropriate quoting. At this point, running under a debugger, with a command line for specific options, a crash of the binary should give you some clue about the problem by checking the stack backtrace to see what function is failing. It is highly doubtful you will be finding problems in the source code for the crashes. The almost universal cause for failure are the data inputs to the binaries. For example, violations of the SQL structures expected from the database tables. Missing data files in the /gbdb/ hierarchy, and so forth. If you are developing code for special track displays, the most common form of problem is a memory violation while using some of the specialized structures, hash lists, etc. Your stack backtrace will usually highlight these situations. In order to determine the URL being used by the browser CGIs to pass to in the debugger, you need to force the browser to use GET http requests rather than POST. Try adding &formMethod=GET to an URL. Not all forms pay attention to that input, but when they do it generally looks like this: hPrintf("<FORM ACTION=\"%s\" NAME=\"mainForm\" METHOD=%s>\n", getScriptName(), cartUsualString(cart, "formMethod", "POST")); If you add &formMethod=GET and a subsequently fetched form is still posting, you might need to alter the "<FORM..." statement to use the cartUsualString. The hg18 hgTracks config page generates a GET URL that is too long for FireFox, so after debugging hgTables, you will probably want to add &formMethod=POST to an URL (or clear cart, load session etc). One thing that does not work with GET is "upload file" inputs. # HTTPS Certificate Check for Verification Settings are: abort warn log none # currently the default is log httpsCertCheck=log # domains to whitelist, skip cert checking, space-separated list httpsCertCheckDomainExceptions=somedomain1.com somedomain2.edu Notes on security The Genome Browser is a complicated piece of software of more than 2 million lines of code that have been developed over 20 years. There is a risk of bugs and security issues, and we take that seriously. At UCSC, we run a security scan on the Genome Browser once a month and pay close attention to the results. Frequently, however, the scanners complain about things in the Genome Browser that are not actually problems. The Browser is designed for significant flexibility when accessing and sharing data. As a result, security scanners often complain about potential vulnerabilities that are actually intentional features: - The Genome Browser draws remote genome annotation files that are streamed on the fly through https. This means that parts of these files appear in our user interface. Various features of custom tracks and track hubs specifically reference external file via URLs that the user can define, and we place no restrictions on these URLs for various reasons. These features predate the introduction of CORS to the internet protocols by at least five years, and the servers from where these files are loaded by the Genome Browser do not need to specifically allow the requests. As a result, this is not a security exploit but an intentional feature, similar to a webservice that checks files entered via a URL (e.g. a robots.txt analyzer) and then shows the content of the URLs on their web pages. Security scanners will find these features and flag them as SSRF exploits (Server-side request forgery), but they are a result of our features, not a problem in our software, and not a problem in the context of our public web servers. - Conversely, when running a local Genome Browser at your institution, do not assume that the Genome Browser web server can be granted access to internal files that should not be shared with the outside world. The Genome Browser is not a secure gateway, and anything that the Genome Browser can access may be displayable by any Genome Browser user. For example, if you restrict internal files to access from only the Genome Browser IP address, as with most other servers without dedicated security layers, assume that these can be accessed by anyone who is using the Browser. We strongly advise you to constrain access to and from your local mirror of the Browser if this is a concern for you. - On our webservers at UCSC and your own internal mirror, even though the data from external URLs is shown, and even though the HTML tags are interpreted, Javascript in these pages is not executed, so an XSS exploit is not possible through these. We use CSP and a nonce to block Javascript injection in these pages. CSP/nonce is a system that many security scanners do not recognize. You can disregard XSS warnings from the security scanners and refer your IT department to our CSP declarations in the HTML headers unless you think that you found a case where it is possible to get through the CSP/nonce system. Feel free to send us the security report from your scanner when in doubt. - Our CSP/nonce system also secures the Javascript libraries. We use customized jquery libraries from the mid-2000s. Some of these are outdated, so some security scanners deem them unsafe, but our use of CSP and a nonce means that possible XSS problems cannot be exploited. The security scanners also cannot detect that. To reduce these warnings, we are in the process of upgrading our Javascript dependencies where possible. Proxy support net.c now has support for http(s) proxy servers which may be required by some installations to get through the firewall to external resources such as (but not limited to) for example bigWig or bigBed data via custom track bigDataUrl. One must add the setting "httpProxy", "httpsProxy", "ftpProxy" to hg.conf httpProxy=http://someProxyServer:3128 httpsProxy=http://someProxyServer:3128 ftpProxy=ftp://127.0.0.1:2121 If the proxy server requires BASIC authentication httpProxy=http://user:password@someProxyServer:3128 httpsProxy=http://user:password@someProxyServer:3128 where user and password may need URL-encoding if they contain special characters such as ":" and "@". If you wish to exclude domains from proxying, create a comma-separated list of domain-suffixes. If a domain ends with an entry from this list, the proxy will be skipped. noProxy=ucsc.edu,mit.edu The httpProxy and httpsProxy URLs should use http protocol, not https. One reason for this is that https sessions would end up doubly-encoded. If you are debugging your proxy configuration, you can use this hg.conf setting to turn on logging to stderr. logProxy=on It is not meant to be left on in production. Your proxy server should have its own logging features. net.c also responds to environment variables http_proxy, https_proxy, ftp_proxy, no_proxy and log_proxy. Support for cloud URLs The genome browser supports cloud URLs by allowing a mirror to configure a command to resolve these URLs to http or https URLs. For example, URLs with the gs, s3, or drs scheme would be resolved to https: URLs. This may involve conversion to signed https URLs. The browser caches the results under the original URL and will call the resolver command again if the signed URL has timed out. URL resolution is enabled by defining following variables in hg.conf: - resolvProts: A comma-separated list of URL protocols, without the colon, to resolve with the specified command. - resolvCmd: The path to the command to resolve the matching URLs. It may include space-separated arguments that are passed on to the command unchanged. The URL to resolve is passed as the last argument. The command writes the resolved http/https URL to stdout. If an error occurs, the command should write an error message to stderr and exit with a non-zero status code. For example: resolvProts = gs,drs resolvCmd = /var/www/tools/urlResolver /var/www/tools/config The UDC local cache directory The udcCache allows tracks that are either installed tracks or custom tracks of the above mentioned types to cache data that they have already fetched via URL. This allows data to reside elsewhere and only download the parts needed on demand. The datablocks are usually compressed and have an efficient random access index. They are accessed from a remote location via URLs such as HTTP, HTTPS, FTP. - udcCache means URL-Data-Cache - BBI files use the udcCache. - BBI means Big Binary Indexed and includes file types such as BigBed (.bb) and BigWig (.bw). - UCSC BAM file support may also use the udcCache By default, udcCache stores files in /tmp/udcCache However, you may include the following in your hg.conf and then let your regular trash cleaning scripts clean out the old udcCache automatically as well: # directory for temporary bbi file caching udc.cacheDir=../trash/udcCache Notice that this path is relative to your cgi executable directory which is the current directory when the cgi starts up. On some systems this directory is called cgi-bin/. Activating CRAM support for the Genome Browser. The UCSC Genome Browser is capable of displaying tracks from both the BAM and CRAM file formats. While BAM tracks provide all of the required data within the file, however, CRAM tracks depend on external "reference sequence" files (see http://www.ebi.ac.uk/ena/software/cram-toolkit for more information about the CRAM format). A bit of information on how the Browser works with these files is included below. For installation instructions, skip to the numbered steps at the end of this file. The directory that Genome Browser CGIs check for CRAM reference files is set with the cramRef setting in hg.conf. For example, the following setting is used on our production servers: cramRef=/userdata/cramCache When loading tracks from the CRAM file format, CGIs will look for reference sequences in that directory. The filename of each reference sequence should be the MD5 or SHA1 checksum of the reference sequence as described at http://www.ebi.ac.uk/ena/software/cram-reference-registry. If a CGI is unable to find the reference sequence file for a CRAM track, it will next check the cramRef/pending/ directory to see if a request for that reference sequence has already been made, and the cramRef/error/ directory to see if a previous attempt at downloading that reference sequence resulted in an error. If none of those files are found, the CGI will then create a request file in the cramRef/pending/ directory. The name of the request file will be the MD5 or SHA1 sequence checksum, as specified in the CRAM data file. The contents of the request file will be the URL to download that reference sequence. A separate tool can then be used to download reference sequences listed in the pending/ directory and place them into cramRef/. Steps to set up CRAM track support: 1. Add the hg.conf setting cramRef. The value should be the path (relative or absolute) to a directory where CRAM reference sequences are stored. 2. Inside the cramRef directory create subdirectories called "pending" and "error". The apache user must have read/write permissions for the pending/ directory, and at least read permissions for the cramRef/ and error/ directories. If you plan to manually load all CRAM reference sequences for your tracks into the cramRef directory, track support is now complete. If you prefer to have reference sequences automatically downloaded and placed in that directory (e.g., for user-submitted custom tracks), continue to step 3. 3. Add a cron job to run a script that parses files in the cramRef/pending/ directory, downloads the corresponding reference sequence files, and places those sequence files in cramRef/. Error messages during file retrieval should be placed in cramRef/error/. An example script is provided in this repository at kent/src/product/scripts/fetchCramReference.sh. The account that runs this script must have read/write permissions for the cramRef/, cramRef/pending/, and cramRef/error/ directories. Using FreeType font support for anti-aliased text The Genome Browser can use a set of hard-coded Type1 fonts for text display in hgTracks. The fonts are NOT distributed as part of the Genome Browser files but they exist natively on the UNIX systems we use in our production environment. To support FreeType font drawing on your system you may need to install these files if they are not already present. On most systems, they exist in one of three directories: /usr/share/X11/fonts/urw-fonts /usr/share/fonts/default/Type1 /usr/share/fonts/type1/gsfonts To turn on the FreeType font rendering you need to add this line to your hg.conf file: freeType=on By default, the Genome Browser looks in the directory /usr/share/fonts/default/Type1 for the font files it expects to find. If you have these fonts in a different directory, you need to add this line to your hg.conf file: freeTypeDir='<apache accessible directory path with fonts in it>' The names of the font files that the Genome Browser expects and their md5sums are in the following table: 5624bc04bbd862287bda3c44b15e90d6 a010013l.pfb 6883f53d0d140f972480f179bb5e4e3b a010015l.pfb db21f79bac990fa7e3de6d4cc6f54020 a010033l.pfb c3d0c3af6bc183010204ca923fb98e83 a010035l.pfb 33265074c9c45c356f90de9cc47259b3 n019003l.pfb 9d07b8658622c2e0127628c87fdb9661 n019004l.pfb df59ecc7bd232d2355786fd644f13baa n019023l.pfb 9f2d4377d9eea95dfdfa705c83a39464 n019024l.pfb 8f75382bd2620a20aaeb06ab8c591e8f n019043l.pfb 28bb40d1700909b2509e2125a1963e2c n019044l.pfb adf4d5565bec1170f0a5810c6984ff55 n019063l.pfb 57e1a49c39a546c2883375df12111816 n019064l.pfb 84e347c88eff2e77ed40b020b457bafe n021003l.pfb 8721c1175d907d7d484bdaa91d4533ad n021004l.pfb eef0367afaa10b929e528c40c980b941 n021023l.pfb 7af41ff3536fadcdc27eb9e5cc1c32d3 n021024l.pfb db95b702b81c12df1f91d15c8a6c3191 n022003l.pfb e97c9af68414fabe157af711b7691df7 n022004l.pfb f13fb8e581426a1ffa5fc12b10ad0f34 n022023l.pfb f3cd9244150f086434b62d5c5bb559b0 n022024l.pfb 30aef717b7c68a6b6b7760b764fbd01c z003034l.pfb Building the kent source tree. In some cases, you may have to modify the source code itself. In these cases, you would need to build all the tools and CGI programs from the source code on your machine. There are scripts available in the git repo src/product/scripts/ directory that can run all of these steps for you with the script: scripts/kentSrcUpdate.sh See the instructions in scripts/README If you have the kent git repository cloned: 1. Check your shell environment. It should contain a MACHTYPE variable. Typical values, for example: i386 i686 sparc alpha x86_64 ppc If there is none, set it to indicate your CPU architecture, for example, MACHTYPE=i686 export MACHTYPE You need to do this if your existing MACHTYPE is a long string such as: i686-pc-linux-gnu since this is used as an argument on the command line to gcc which will not work with the - signs in the strings. Typical values: i386 i686 x86_64 sparc ppc To determine the machine cpu type of your computer, try one of these uname command options: $ uname -m $ uname -p $ uname -a Remember to set this MACHTYPE in your .bashrc or .tcshrc home directory environment files so it will be set properly at your next login. The browser code has not been tested by UCSC outside a CentOS Linux on Intel or AMD Opteron environment. Other users do report successful operation on other systems. Depending upon what libraries you have installed locally, you may want to set environment variable USE_SAMTABIX. See also: http://genomewiki.ucsc.edu/index.php/Build_Environment_Variables 2. Create a directory where binaries will be moved to during the build of the source tree: $ mkdir -p ~/bin/${MACHTYPE} 2a. ALTERNATE PATH: If you are going to do a full build anyway, skip this and proceed straight to step 3 below. Otherwise, to make a minimal utility build without MariaDB: There are some utilities that depend only on jkweb.a and not jkhgap.a which means they can be compiled without needing MariaDB client installed. To make a utility like pslCDnaFilter without installing MariaDB client: # create jkweb.a cd kent/src/lib make # create stringify utility required by some makefiles cd kent/src/utils/stringify make # create pslCDnaFilter utility program cd kent/src/hg/pslCDnaFilter make Proceed similarly for any other such utilities. You are done and can stop here. 3. Create the following shell environment variables: NOTE: As of mid-October 2013 the makefile build system in the source tree will attempt to configure MYSQLINC and MYSQLLIBS with the mysql_config command. To use that automatic configuration, eliminate any MYSQLINC and MYSQLLIBS definitions from your shell environment and make sure mysql_config can be found in your PATH. This option is not perfect and may not function correctly. It usually will not use a static linked library which can lead to a known issue on the Mac OSX. See 'Known Problems' item 10 below. In the case the automatic configuration does not function, you can set these variables in your shell environment to override the automatic configuration: MYSQLINC=/usr/include/mysql MYSQLLIBS="/usr/lib/mysql/libmysqlclient.a -lz" export MYSQLINC MYSQLLIBS Your setting may vary depending upon where your MariaDB is installed. The above example is typical. If your system does not have this set of include files or this static client.a file, you may need to install the mariadb-devel package to obtain the MariaDB development environment. With that package installed, this command: mysql_config --libs will display the appropriate libraries to link with for your system configuration. And: mysql_config --include will display the appropriate MYSQLINC directory. The -lz requires a link to the libraries installed in the zlib-devel rpm. 3a. Required SSL support: In order for the libraries to be able to use SSL, for instance to support fetching HTTPS URLs, install openssl. We are currently using these packages: openssl-1.0.1e openssl-devel-1.0.1e SSL is also useful for accessing bigWig and bigBed over HTTPS. The directory on your server should be found automatically. 3b. In the source tree, perform the following: $ cd kent/src $ make libs At this point, you can build utilities in other parts of the source tree if that is your goal. Go to the directory of the utility command you want to build and run a 'make' in that directory. The resulting binary will be placed in \(HOME/bin/\)MACHTYPE 4. To continue building the CGI binaries $ cd kent/src/hg $ make compile $ make install DESTDIR=/destination/prefix Where "/destination/prefix" is your choice, and this will install the cgi binaries in: /destination/prefix/usr/local/apache/cgi-bin If your cgi-bin directory is something different than /usr/local/apache/cgi-bin then use the CGI_BIN variable, e.g.: make install DESTDIR=/var/www CGI_BIN=/cgi-bin \ DOCUMENTROOT=/var/www/html You can save yourself time and trouble if your Apache is somewhere other than at /usr/local/apache by creating that directory and making symlinks to your actual apache directories. For example: mkdir /usr/local/apache ln -s /var/www/cgi-bin /usr/local/apache/cgi-bin ln -s /var/www/html /usr/local/apache/htdocs ln -s /var/www/cgi-bin-${LOGNAME} /usr/local/apache/cgi-bin With those symlinks in place, a simple 'make cgi' can be used instead of the 'make compile; make install DESTDIR=...' business. If your apache DocumentRoot is something different than /usr/local/apache/htdocs then use the DOCUMENTROOT variable. This value should be a full path and should agree with the browser.documentRoot setting in hg.conf; see the section "MariaDB Setup" for more details. $ make install DESTDIR=/destination/prefix CGI_BIN=/cgi-bin/path DOCUMENTROOT=/usr/local/apache/htdocs to install binaries in "/destination/prefix/cgi-bin/path" [NOTE: These 'make' commands assume gnu make is being used] 5. After source tree updates, the make sequence is: $ cd kent/src $ make clean $ make libs $ cd hg $ make compile $ make install DESTDIR=/destination/prefix ------------------------------------------------------------------------ Known problems: 1. A compile/link of a utility has complaints about undefined symbols such as: bind, accept, listen, gethostbyname, socket, setsockopt, connect, inet_pton, inet_ntop or h_errno To fix this, you need to add '-lsocket -lnsl' to your MYSQLLIBS environment variable. This is typically the case on Solaris systems: MYSQLLIBS="/usr/lib/mysql/libmysqlclient.a -lsocket -lnsl -lz" 2. The build fails immediately in the first src/lib/ directory with the compiler issuing a warning about unused variables. Some newer versions of gcc issue these warnings and the src/inc/common.mk file specifies -Werror which causes it to exit on these warnings. Either remove the -Werror specifications in src/inc/common.mk or add the -Wno-unused-variable to instruct the compiler to allow these warnings without an exit. 3. The build fails during a link of any program under the src/hg/ hierarchy with an error something like: undefined reference to 'SSL_CTX_free' undefined reference to 'ERR_get_error_line_data' undefined reference to 'SSL_read' undefined reference to 'SSL_get_error' undefined reference to 'SSL_write' This error is due to your MariaDB libraries have been compiled with SSL functionality enabled. To fix this build problem, add '-lssl' to your MYSQLLIBS environment variable to satisify these SSL library functions. 4. Build fails on Macintosh with an error: aliType.c:5: warning: 'rcsid' defined but not used make: *** [aliType.o] Error 1 The OSTYPE environment variable needs to be set to "darwin". If your shell is bash, it is a shell local variable instead of an environment variable as with tcsh. To avoid this error, place an export statement in your $HOME/.bashrc environment: export OSTYPE This causes the OSTYPE variable to be recognized during the make and the conditional statements in src/inc/common.mk will include the -Wno-unused-variable option in the build. 5. The build fails with an error: gifdecomp.c:274: error: call to function 'gif_out_line' without a real prototype This happens with newer versions of the gcc compiler. To avoid the build stopping at this point, eliminate the -Werror argument from the src/inc/common.mk file. We will fix this error as soon as we get our hands on a newer version of the gcc compiler. 6. The build fails with an error of this type: pipeline.c: In function waitOnExec: pipeline.c:351: error: ignoring return value of read, declared with attribute warn_unused_result This happens with newer versions of the gcc compiler. To avoid the build stopping at this point, eliminate the -Werror argument from the src/inc/common.mk file. We will fix this error as soon as we get our hands on a newer version of the gcc compiler. 7. Build fails complaining about missing png.h file: pngwrite.c:7:87: error: png.h: No such file or directory Find out where your png.h file is and set PNGINCL to something like: export PNGINCL="-I/opt/local/include/libpng14" pointing to the directory where png.h exists. 8. Build fails at the point of building a CGI binary with the error: Undefined symbols: "_png_write_png", referenced from: _mgSaveToPng in jkweb.a(pngwrite.o) ... other png symbols ... You need to find out where your libpng.a is located. For example it may be in /opt/local/lib/libpng.a When found, use the PNGLIB variable to specify it, for example: export PNGLIB=/opt/local/lib/libpng.a or equivalent: export PNGLIB='-L/opt/local/lib -lpng' 9. Build fails with complains about missing functions dlclose, dlopen: /usr/lib/x86_64-linux-gnu/libmysqlclient.a(client_plugin.c.o): In function 'add_plugin': (.text+0x1ed): undefined reference to 'dlclose' /usr/lib/x86_64-linux-gnu/libmysqlclient.a(client_plugin.c.o): In function 'mysql_client_plugin_deinit': (.text+0x28b): undefined reference to 'dlclose' /usr/lib/x86_64-linux-gnu/libmysqlclient.a(client_plugin.c.o): In function 'mysql_load_plugin_v': (.text+0x51e): undefined reference to 'dlopen' To add the library that includes these functions, add '-ldl' to the MYSQLLIBS string: MYSQLLIBS="/usr/lib/mysql/libmysqlclient.a -ldl -lz" 10. Mac OSX dynamic link with the MySQL/MariaDB libraries results in the following run-time error for CGI binaries or command line programs: dyld: Library not loaded: libmysqlclient.18.dylib This is a known problem with the dynamic library for MySQL, with potential work-arounds offered: http://bugs.mysql.com/bug.php?id=61243 Or, you can set the MYSQLINC and MYSQLLIBS shell environment variables to the static MySQL library as mentioned in Step 3 above. 11. Genome browser issues error: "PDF format not available" when trying to export to pdf. The browser can't find the command 'ps2pdf' which is usually found in /usr/bin/ps2pdf and is installed with the 'ghostscript' package. Install the 'ghostscript' package and verify ps2pdf is in: /usr/bin/ps2pdf Adding a track hub to your hubPublic table so it appears under My Data > Track Hubs By default, your browser mirror only comes with our public hubs. You may want to remove these and replace them with the hubs in your institution. One idea could be to have one hub per research group or department and users connect to them on My Data > Track Hubs. Use hubPublicCheck to create the SQL command to add the hub: hubPublicCheck hubPublic -addHub=https://genome.ucsc.edu/goldenPath/help/examples/hubDirectory/hub.txt Output is: insert into hubPublic (hubUrl,descriptionUrl,shortLabel,longLabel,registrationTime,dbCount,dbList) values ("https://genome.ucsc.edu/goldenPath/help/examples/hubDirectory/hub.txt","", "My Hub's Name", "Name up to 80 characters versus shortLabel limited to 17 characters", now(),1, "hg19,"); Then we run hubCrawl to get the searches to work (most mirrors do not need this). See https://genomewiki.ucsc.edu/genecats/index.php/Public_Hub_QA for more details.