Como remediar Spectre y Meltdown en ambientes de vSphere / VMware.


La última semana se ha hablado mucho de Spectre y Meltdown, vulnerabilidades de procesadores modernos que cuando son explotados, permiten que programas puedan obtener información que otros programas han almacenado en memoria como passwords, emails, fotos y otra información personal que normalmente no deberían poder ser accedidos por otros procesos corriendo en el mismo servidor…..hasta ahora.


Puedes leer más acerca de Spectre y Meltdown aquí: https://meltdownattack.com/

Estos bugs están catalogados bajo CVE-2017-5753, CVE-2017-5715 y CVE-2017-5754 (Spectre y Meltdown)

Estas variaciones de bugs han sido categorizadas como unas de las más grandes y masivas para la industria ya que afecta procesadores generados en la última década, es decir, millones de sistemas en todo el mundo, desde computadores personales, celulares y servidores en la nube, estos últimos considerados los más susceptibles ya que en la mayoría de los casos son compartidos por diferentes clientes and ambientes virtualizados. 


A pesar de que ciertas fuentes conocían esta vulnerabilidad hace unos meses, el tema se hizo público el pasado 3 de Enero (Puedes leer mas aqui:  https://googleprojectzero.blogspot.ca/2018/01/reading-privileged-memory-with-side.html) y como era de esperarse, se convirtió en urgencia para compañías de hardware y software desarrollar parches para remediar en lo posible dichas vulnerabilidades. Aunque inicialmente se escuchó que habría que reemplazar cada uno de los procesadores físicos, lo cual sería algo que tomaría años; rápidamente, la combinación de actualizar todos los componentes en los servidores (BIOS/CPU Microcode, Hypervisor, VM Guest e incluso versión de hardware en el caso de VMware) al parecer serían suficientes para estar protegido. A juzgar por los blogs y documentos oficiales, al menos de VMware, si se actualizan todos los componentes, se estará a salvo - Esperemos que así sea.

Aquí dejo descripción de los diferentes pasos requeridos al día de hoy, si buscas remediar tu ambiente de vSphere:


1) Actualizar vCenter Server a las siguientes versiones; dependiendo cual de las ultimas 3 aún bajo soporte uses:
     vCenter 5.5 U3g

        2) Actualizar los hosts ESXi a las siguientes versiones:

  ESXi 5.5
 ESXi550-201801401-BG  - Este parche actualiza el Hypervisor-Assisted Guest Mitigation (el cual es necesario para que al después de aplicar updates de los sistemas operativos dentro de tus VMs, las nuevas características de la CPU del host puedan ser vistas y correctamente utilizadas por las máquinas virtuales) y CPU Microcode que esencialmente reemplaza parte de código BIOS para el CPU pero no reemplaza el BIOS por completo.
  ESXi 6.0
ESXi600-201801401-BG – Este parche corrige el Hypervisor-Assisted Guest Remediation
ESXi600-201801402-BG – Mientras este, trae los cambios necesarios para el Microcódigo de CPU
  ESXi 6.5
ESXi650-201801401-BG – Este parche corrige el Hypervisor-Assisted Guest Remediation
ESXi650-201801402-BG – Mientras este, trae los cambios necesarios para el Microcódigo de CPU
Los bundles que actualizan Microcódigo del CPU son opcionales y dependerá de si ya aplicaste el BIOS completo que debe ser proveído por la compañía de hardware.
    
     3) Asegurarse que todas tus máquinas virtuales están usando Versión de Hardware 9 como mínimo, de lo contrario, las nuevas características de CPU proveídas por el parche de ESXi no tendrá efecto y tus máquinas no estarán protegidas.


     4) El siguiente paso es instalar los parches para tus sistemas operativos que cubran vulnerabilidad CVE-2017-5715. Estos suministrados por Microsoft, RedHat, etc.  


     5) Por último, necesitaras apagar y reiniciar tus VMs, full power cycle. Bien sea después de actualizar la versión de hardware (9 como mínimo) o después de moverlas a hosts ya actualizados.



Algunas notas para resaltar y tener en cuenta:



El vCenter Server Appliance (VCSA) obtendrá un update que cubrirá su propio sistema operativo; así que es recomendado estar pendiente de cuándo saldrá. La información será actualizada aquí: https://kb.vmware.com/s/article/52264

Si tus ESXi hosts no fueron actualizados con los parches ESXi550-201801401-BG, ESXi600-201711101-SG o ESXi650-201712101-SG que fueron publicados el pasado mes y detallados en el Security Advisory VMSA-2018-0002 (https://www.vmware.com/us/security/advisories/VMSA-2018-0002.html), podrás omitirlos y simplemente instalar los nuevos; estos updates son acumulativos.

Si tienes maquinas virtuales que no están a nivel de virtual hardware 9, las nuevas características agregadas al CPU no serán presentadas y así hayas aplicado updates del sistema operativo, estas no se harán efectivas hasta que TODO este actualizado; de lo contrario tus VMs estarán desprotegidas.

La versión de Hardware de cualquier Virtual Appliance (sean de VMware o terceros) no deben ser actualizadas manualmente; debes esperar que las nuevas versiones sean suministradas y vengan con versión 9 o mas alta.   

Si actualizas el sistema operativo antes de aplicar los parches de ESXi (Hypervisor-Assisted Guest Mitigation) las VMs requerirán ser apagadas e iniciadas de nuevo.

Si tienes curiosidad, el update a vCenter solo modifica código, no hace cambios a la base de datos y es muy rápido de aplicar.

Recuerda que así instales el CPU Microcode que VMware suministro, debes instalar el BIOS completo cuando este este disponible ya que actualizara todos los componentes del BIOS.



===========================

Comparte, pregunta, agrega o reta mi opinión aquí expuesta. Seria genial escuchar opiniones de quienes lean y ojala se beneficien de esta información; tambien de quienes puedan estar en desacuerdo. 


Recuerda que la información aquí presentada, como en cualquier área de mi blog, es mi opinión y no representa la de mi empleador.






VMworld 2017 - First day

Although VMworld doesn’t officially start until Sunday or depending how you look at it, Monday; the whole weekend leading to it is an amazing opportunity to get familiar with the huge Mandalay complex, find out where the different places are located and of course one of the best parts of the show… meet fellow technology enthusiasts. This is one of the most valuable things of attending the conference in my opinion.

A day and a half in and I have probably already met 20+ new vFriends from the community and have had excellent conversations about the cloud, infrastructure, automation and even great personal experiences.

One of the anticipated moments for me was to meet Al Rasheed who won the conference pass I was able to donate to the vCommunity last month; Al had never attended VMworld before and just 24 hours in, he is thrilled and having a lot of fun. He has already taken a picture with VMware CEO Pat Gelsinger.




This time around it is also very important to me to be here to meet my new team and continue learning about the TAM role and the organization in general. On Sunday, I attended the TAM Reception, talked and learned a bunch from TAM customers, their feedback about the program and how much they value the relationship.






The first official party of VMworld 2017 was the VMUG Party and as expected, it didn't disappoint; the food was great and the band was simply amazing. They rocked the house and gave us an awesome time to start the week with the right foot.



Getting together with my vBrownBagLATAM friends is always a pleasure and we had a chance to have dinner after the VMUG party. As usual we had a great time.




I am looking forward to the coming days being as amazing as the first one, to the different announcements VMware will have, specially related to the partnership with AWS and to meeting new friends in the community and create lasting relationships.



Opportunity knocks


-CHANGE-

"Is a necessary part of life. Be willing to surrender what you are for what you could become."


After three amazing and worthwhile years as a Sr. Infrastructure Engineer leading the VMware platforms at Bain Capital, I will be leaving at the end of this week to take on a new and exciting opportunity for my career.
Not only am I leaving the company but the city of Boston and the State of Massachusetts where I have grown professionally and have met truly amazing people.

I will be relocating 500 miles south to the Washington Metropolitan area and join VMware as a Sr. Technical Account Manager.

This will certainly be a life changing event for me and my family, but it is one that we're all looking forward to positively and with great enthusiasm.

There is a lot we are giving up, but in my mind, is not what we are leaving behind but the possibilities of what we could gain; in my mind, I see a great future for me and my loved ones and hopefully a lot less snow to shovel during winters. 😊



VMworldbound Contest

After some consideration, here is how the winner of the #VMworldbound contest will be selected.

On Tuesday August 1st, seven finalists will be picked by William Lam and myself; these people will be notified and expected to confirm no later than Wednesday morning at 9am EST that if they win the VMworld Pass, they will be able and planning to attend, for sure.
The idea and point of this giveaway is to make sure this pass is not wasted and provide someone in the community the opportunity to enjoy a great conference. 
If any of the finalists can't make it for whatever reason, another person not initially picked will enter the "Selected 7".

Announcements for finalists will be made on Twitter and/or via the medium the finalist used to submit her/his entry. The soonest a confirmation of attendance is received the better.

On Wednesday night I will be broadcasting on Periscope the actual drawing and selection of the winner. The names of finalists will be placed in a bag/hat/box (or whatever container I find) and a lucky one will be pulled. 

If you want to be notified and watch the live broadcast, even if you're not participating, you'll need to enable account notifications on Twitter.

Give this a try, you never know what the future may bring.

Good luck to all participants!




Free VMworld Pass Contest

Would you like to go to VMWorld and don't have a pass? 
If your answer is yes, read on... You may be the lucky winner of a free registration code for it.

I want to give someone the possibility of attending VMworld 2017 in Las Vegas next month, so I'm giving away a validated registration code for the big conference to a lucky person in the vCommunity and hopefully make a difference in that person's career.

Ideally, the gal or guy would be someone who has not attended VMworld before, although having attended in the past doesn't disqualify you from participating :) 

Would like to see a person who is enthusiastic about virtualization, eager to learn and to take full advantage of everything this event has to offer.


Having attended my first and only VMworld last year made me realize the immense value this conference presents if you take advantage of its opportunities; you get to see great stuff on the different breakout sessions, practice and learn the latest products and/or features from VMware in the Hands-on-Labs, meet amazing and smart people and have tons of fun in the process.

When the additional registration code became available to me this week, I thought of contributing it to somebody who may not have the chance of attending and giving that individual the opportunity to experience the awesomeness that I enjoyed last August.

So, for a chance to win, comment on this blog or use hashtag #VMworldbound on Twitter and list the reasons why you would like to attend VMworld 2017. Make sure you tag @lamw and myself @j_kolkes if using Twitter.


Community leader William Lam will be picking and announcing the winner on August 2nd.

The only requirements are:
  • You want and are able go to VMworld2017 in Las Vegas.
  • You can cover flight and hotel expenses. They're not included in this giveaway.
  • You promise to meet with William and myself for a drink and picture while at the event ;)

Update: Click here to learn how the winner will be selected.




.NEXT, what I'm expecting.

Back in December 2016 the vBrownBag Community sponsored by technology companies, gave out prizes to its followers; some prizes included brand sunglasses, drones and other cool stuff; I don't remember the complete list but there were many. One of the most highly valued prizes was a pair of passes to the Nutanix .NEXT Conference in Washington DC. I was one of the lucky winners, so I'm getting ready to take advantage and attend the 3 day event this week for the first time. 




As a virtualization enthusiast, I enjoy learning different technologies, products and solutions; unfortunately, Nutanix isn't a solution I have used in the past, not because I don't want to, but because I haven't had a chance to really kick the tires and dive into it a bit deeper. I understand Nutanix's solutions based on presentations and documents read, but the way my brain absorbs the most is by doing or in this case by actually using and navigating the application; so I am anticipating to hit the Nutanix Labs hard this week, ask many questions during the different sessions and talk with NTC's and hopefully become a more knowledgeable HCI / Nutanix / Acropolis / Prism / Enterprise Cloud user myself. 
Of course this being a large conference where many of the community users attend, I also expect to see my old friends and make new ones! 





These types of conferences are a great opportunity to network, meet other professionals and of course learn a bunch; I'm looking forward for all these expectations to be fulfilled during the .NEXT Conference this June 28th through 30th. If you see me there, say hi 😉




Zerto, my Notes and Thoughts

Earlier this year, I was involved on a POC for Zerto Virtual Replication in a VMware environment and took some notes of things that I liked and found useful about the product and the way it works.

I wanted to share here some of the things I have learned, for my own reference and obviously for anyone out there who is starting with the product and may find these notes useful. A disclaimer here though: Things you read here could have been misinterpreted or misunderstood by me and you should research and use Zerto's Technical Documentation if you plan on implementing it in your production environment.

Overall, I really like Zerto; it is intuitive and simple to use, yet very powerful and complete application that will allow you to protect virtual machines with RPO in seconds and give you very convenient features.

Let's review some basic acronyms and components you need to be familiar with:
Zerto Virtual Manager or ZVM: It is the central management interface installed on a Windows server, it allows you to manage all the DR tasks related to your source and target sites. You need one ZVM per vCenter Server.
Virtual Replication Appliance or VRA: This is the appliance deployed to each one of the hosts in the cluster where the VMs you intend to protect reside, as well as in the target hosts. These appliances manage the actual replication of data from source to target site. VRA's run Debian Linux for operating system.
VPG: Virtual Protection Group; it is the grouping of servers that replicate with the same parameters or settings; often used to group servers of the same application stack, so they can be tested and recover together. Important as when you failover to a Checkpoint, they are all consistent.

From my personal point of view and own experience I will list some of the features and cool options in no specific order.

The installation process: It cannot be simpler. Installing Zerto is straightforward process; you will need one Windows Server to install the software and link it to its dedicated vCenter Server. The software requires a minimum of 4GB of free space. The installation wizard offers two options; one is the “Custom installation” which will give you the opportunity to select a specific account to run the Zerto Virtual Manager service, the ability to choose an external or embedded database. The “Express installation” will use embedded database and run its service as Local System. Regardless of the option, you will need to enter the FQDN of your vCenter, an account with permissions and a Site Name. From the installation wizard you can choose to participate in the Online Services and Zerto Mobile Application which will give you access to Zerto Analytics which is a great new tool that's expanding. At the end of the wizard communication and credentials to vCenter are validated; if there are any issues, a warning will display. Installation completes within 5 minutes.

Logging in for the first time: You access Zerto from a browser on port 9669 (https://DR-vCenter.kolkes.com:9669/zvm).
You need to provide a license key when you first login, so you either enter a the key manually or pair with another site that is already licensed and running.
Its HTML5 interface is clean and very responsive, you see multiple tabs where you configure different things but one thing I found useful in this product is that you can access and initiate many tasks from various places in the UI. 
On a brand new installation there will be pop-up messages that will guide you through finalizing the setup and things you need to do in order to start protecting your VMs.

Main Zerto management screen


The tabs in the UI are intuitive but here is a quick summary of them: