How to open booking form in popup (modal window) ?

You can use the code like this for opening your booking form in poup (modal window):

<a href="javascript:void(0)" onclick="javascript:jQuery('.bookingFormModal').modal('show');" class="btn button">Booking form in popup window</a></p>
<div class="wpdevbk">
<div class="modal bookingFormModal" style="display:none;width: 700px !important;height: 1000px !important;">
<div class="modal-header"><a class="close" data-dismiss="modal">& #10006;</a></p>
<h3 style="margin-top:-27px;">Booking Form</h3>
<div class="modal-body"> [booking type=4 nummonths=2] </div>

where [booking type=4 nummonths=2] - shortcode for your booking form.
Probably additionally you will be need to customize some CSS classes and CSS styles relative to your actual theme.