[olug] Request for feedback from users of KVM-based virtualization in RHEL Server 6

[Carl Lundstedt]

Robert,

Have you considered asking the HCC staff for help?  We have three 
applications specialists that can assist you in designing workflow and 
coding analysis.  I'm not an app specialist, so I can't really answer 
your query on the benefits/drawbacks of virtualization (I'm a 
physicist/system admin).  We also have very talented and knowledgeable 
students that are available for researchers.

Is your application/analysis map-reduce friendly?  Such an analysis 
would be right in hadoop's wheel house (large dataset, high 
parallelization).  There used to be a smallish (100TB or so) hadoop 
install on the Lincoln HCC cluster Prairiefire, which you would have 
access to.  I don't know PF's hadoop status, but there's sizable hadoop 
install on the CMS cluster red in Lincoln (1.6 PB) that we may be talked 
into utilizing for non-CMS U of N general usage, especially if your 
workflow could be submitted via our globus grid interfaces (again, our 
staff can help you there).  We've shut off the job-tracker on red, but 
we may be willing to rethink having map-reduce jobs come back to test 
the new hadoop awareness being added to condor.

In any case, consider dropping the HCC support list a note and seeing 
what they say.  Perhaps you can get your work done on our resources 
without building any infrastructure of your own.

Carl Lundstedt
UNL/HCC
> Hi everyone,
>
> I joined UNMC's faculty last year, and my research and teaching interests are in bioinformatics and computational biology. Specifically, I focus on the management and analysis of the very large data sets produced by what are known as "next generation" DNA sequencing (or "NGS") instruments.
>
> We perform some computations on some very large distributed memory computer clusters at the Holland Computing Center (HCC) at UNL/UNO and on the TeraGrid. However, very large SMP systems are also useful for many of the compute tasks that we need to perform. By HPC standards we don't have a large budget for compute and storage infrastructure, and we don't want to duplicate resources that are already available to us through the HCC and TeraGrid. What we have settled on is a flexible, low-maintenance, on-premises "private cloud" type approach starting off with just two very large SMP systems each with its own direct-attached storage JBOD array. We can't afford a SAN or NAS storage. These are research computing systems, and will not run software related to UNMC's administrative or teaching missions.
>
> We are dealing with very large DNA sequence datasets (multi-100's of GB to multi-10's of TB), some of which contain DNA sequence data from tissue derived from human subjects, and the movement of these datasets to public cloud computing service providers is thus not really practical logistically and due to regulatory (HIPAA) compliance issues.
>
> KVM-based virtualization in RHEL Server 6 really appeals to us because the VMs we would like to deploy would themselves need to be substantial systems capable of the many different NGS-related compute tasks we will need to be able to perform. We would strongly prefer to not use a bare-metal hypervisor, as for some tasks we do want to use a physical (on-the-metal) host OS. Granted, this is not your typical enterprise virtualization scenario. Nevertheless, I would be very grateful if I might be able to hear some feedback from users of KVM-based virtualization in RHEL Server 6. For example, how does it compare with VMWare VSphere/ESXi? How well does it handle virtualized desktops that might need to deliver graphics-intensive applications? What are the best 10 GbE adapters to use to support virtualization under RHEL Server 6? What are some good open-source or reasonably priced commercial management tools for small-scale on-premises private clouds?
>
> So if you can help, please contact me either through this forum or directly via e-mail (rboissy at unmc.edu) or voice (402-559-4307).
>
> Thanks in advance for your consideration of this request!
>
> Best wishes,
>
> Robert
>
> Robert Boissy, Ph.D.
> Assistant Professor
> Department of Internal Medicine
> University of Nebraska Medical Center
> _______________________________________________
> OLUG mailing list
> OLUG at olug.org
> https://lists.olug.org/mailman/listinfo/olug