Differences

This shows you the differences between two versions of the page.

Link to this comparison view

en:phedex_-_data_transfer_to_sites_without_se [2016/12/05 14:43] (current)
Line 1: Line 1:
 +Last modified: Feb 17, 2014 by Kurca\\
 +\\
 +
 +====== PhEDEx - Data Transfer to sites without SE ======
 +
 +\\
 +\\
 +
 +====== Chapitre 1 : Question - Problem Definition ======
 +
 +I would like to know if PhEDEx can be used to transfer data to centers without an SE.\\
 +We have some storage and computing resources locally which we would like to use for analysis of PAT datasets but without setting up a full T3 center (which I assume means having at least a CE+SE+WN).\\
 +The reason we want to transfer data locally is that we would like to add to the event content some analysis output and store it aside with the original data.\\
 +\\
 +1.\\
 +Your site needs to be officially registered in SiteDB as a T3 to be able\\
 +to transfer data in the PhEDEx Production instance.\\
 +However, you don't need to install a "​full"​ T3 site (CE+WN+SE) to be\\
 +registered as a T3...\\
 +\\
 +2.\\
 +In principle, a User Interface with a large disk would be sufficient to\\
 +install PhEDEx for imports only (I did that on my desktop as part of my\\
 +PhEDEx training :-) ). Let us know the details and we can evaluate the\\
 +registration request. I think there are a couple of T3_US sites running\\
 +in this configuration.\\
 +\\
 +2.\\
 +In practice, a User Interface + a Storage Element with SRMv2 would be\\
 +better. This is also necessary if you ever want to *export* back the\\
 +results of your local analysis.\\
 +\\
 +3.\\
 +I can also confirm that you don't need an SE to run PhEDEx. Our T3 at\\
 +UC Riverside just has a CE+WN. However, it is true that without an\\
 +SE, you can't '​publish'​ your data to let others know that you have\\
 +it. And by others, that also means CRAB. Our users just use the\\
 +local batch system when they want to run on the data that we have\\
 +available.\\
 +\\
 +4.\\
 +Even if you don't have an SE, you still can run the dataset on your disk\\
 +with CRAB or let other users know.\\
 +You can do this by publishing your dataset at your disk on a local DBS\\
 +or borrowing other people'​s local DBS.\\
 +Then, you can put the local DBS instead of global one in your crab.cfg.\\
 +Also, you can even see all the local DBS around the world.\\
 +So, all the means are out there.\\
 +Bockjoo ​
 +====== Chapitre 2 : PhEDEx Installation ======
 +
 +see examples : T3_ES_Oviedo , T3_US_RIVERSIDE
 +<​code>​
 +- Install a recent version of the gLiteUI 3.1/SLC4
 +- Install PhEDEx with the minimal set of agents required (DownloadAgent,​
 +FileExport, FileRemove)
 +- Configure the DownloadAgent to use the SRM backend, the srmcp command,
 +and the srmcp command line option for srmv2 protocol
 +(-srm_protocol_version=2)
 +- Configure the FileExport, DownloadAgent and storage.xml (TFC) to
 +publish a fake '​srmv2'​ protocol that actually maps to
 +'​file:////​yourlocaldiskpath'​
 +- Configure FileExport to publish storage.xml but ignore all sites in export
 +- Configure the FileRemove agent and the FileDownloadVerify,​
 +FileDownloadDelete scripts to use local POSIX commands (ls, rm, etc.)
 +for deletion.
 +</​code>​
 +
  
  • en/phedex_-_data_transfer_to_sites_without_se.txt
  • Last modified: 2016/12/05 14:43
  • (external edit)