Writing VCR Provider Spec Tests

So you’ve just done a refresh with your new provider, congratulations. Now it is time to make sure it continues to work by writing some tests.

VCR

How do you test code which makes API calls without having access to a live system? There is a ruby gem called VCR which “records” HTTP API calls and then plays them back later. This allows us to test code which would normally have to point to a live API without having to stub every single method call.

The results of the recording are stored in the VCR config.cassette_library_dir as YAML files with request/response pairs. VCR will automatically stub the HTTP layer for you, and when it recognizes a request it will replay the saved response for you. This allows for mostly normal operation while running in an isolated CI system.

Configuration

If you generated your provider plugin with the --vcr flag then the VCR configuration should have already been done for you. If not simply add the following to the bottom of your spec/spec_helper.rb file:

VCR.configure do |config|
  config.ignore_hosts 'codeclimate.com' if ENV['CI']
  config.cassette_library_dir = ManageIQ::Providers::AwesomeCloud::Engine.root.join('spec/vcr_cassettes')

  VcrSecrets.define_all_cassette_placeholders(config, :awesome_cloud)
end

The next thing we have to take care of is hiding “secrets”. Since the VCR YAML files will be committed to source control it is critical that private information like passwords do not make it into these files.

VCR handles this with the config.define_cassette_placeholder option. You provide VCR with a string that you want to be replaced, and then what you want it to be replaced with. This allows for hostnames / passwords / etc… to be used when recording the cassette but the values will not be written to the resulting YAML files.

ManageIQ has a pattern to help you with this. By default, the generator created a file named spec/config/secrets.defaults.yml with a username and password.

---
awesome_cloud:
  username: AWESOME_CLOUD_USERNAME
  password: AWESOME_CLOUD_PASSWORD

If your provider uses a different set of secrets, such as access_key and secret_key, you can change the file accordingly as follows:

---
awesome_cloud:
  access_key: AWESOME_CLOUD_ACCESS_KEY
  secret_key: AWESOME_CLOUD_SECRET_KEY

Finally, create a spec/config/secrets.yml file with your real provider secrets. NOTE: This file must not be committed and should be in your .gitignore.

---
awesome_cloud:
  access_key: YOUR_REAL_ACCESS_KEY
  secret_key: YOUR_REAL_SECRET_KEY

And that’s all! The VcrSecrets.define_all_cassette_placeholders line in spec/spec_helper.rb automatically marks everything under the awesome_cloud key as sensitive data.

If you need to manually mark something as sensitive data, then you will need to call config.define_cassette_placeholder. To do so, you can add the following to your VCR.configure block in spec/spec_helper.rb after setting the config.cassette_library_dir. For example, if your provider Base64 encodes the access_key and secret_key into a header, you will want to include something like the following:

  config.define_cassette_placeholder("AWESOME_CLOUD_AUTHORIZATION") do
    Base64.encode("#{VcrSecrets.awesome_cloud.access_key}:#{VcrSecrets.awesome_cloud.secret_key}").chomp
  end

Writing the tests

Now that we have VCR configured it is time to start writing your spec tests. First we will start with the Refresher to test the refresh process of your new provider.

Create a file called spec/models/manageiq/providers/awesome_cloud/cloud_manager/refresher_spec.rb:

describe ManageIQ::Providers::AwesomeCloud::CloudManager::Refresher do
  include Spec::Support::EmsRefreshHelper

  let(:zone) { EvmSpecHelper.create_guid_miq_server_zone.last }
  let!(:ems) do
    FactoryBot.create(:ems_awesome_cloud, :zone => zone).tap do |ems|
      access_key = VcrSecrets.awesome_cloud.access_key
      secret_key = VcrSecrets.awesome_cloud.secret_key

      ems.update_authentication(:default => {:userid => access_key, :password => secret_key})
    end
  end

  describe ".refresh" do
    context "full refresh" do
      it "Performs a full refresh" do
        # Run the refresh twice to catch any record duplication across
        # multiple refreshes
        2.times do
          with_vcr { described_class.refresh([ems]) }
        end

        # Reload the ems record to ensure all associations are up to date
        ems.reload

        # Add your tests here
        assert_ems
        assert_specific_vm
        assert_specific_flavor
        assert_specific_template
        # etc...
      end

      def assert_ems
        expect(ems.last_refresh_error).to be_nil
        expect(ems.last_refresh_date).not_to be_nil
        expect(ems.last_inventory_date).not_to be_nil

        # Update the counts to match what you have in your test environment
        # expect(ems.vms.count).to eq(1)
        # expect(ems.flavors.count).to eq(2)
        # expect(ems.miq_templates.count).to eq(3)
        # etc...
      end
    end
  end
end

With that file created run the specs with the rspec command:

bundle exec rspec spec/models/manageiq/awesome_cloud/cloud_manager/refresher_spec.rb

That should run through and end up creating a file in spec/vcr_cassettes/manageiq/providers/awesome_cloud/cloud_manager/refresher.yml

From now on when you run the specs it will use the results in the file rather than hitting the API directly.

Now fill out the refresher_spec.rb file with more checks to ensure that inventory is collected and associated properly.

Updating the VCR cassettes

Now that you have your specs recorded, what happens if you want to collect something new? For example, if you now want to start fetching floating IPs or Cloud Volumes?

It is simple to re-record your VCR cassette, simply remove the file then rerun the specs against the same environment:

rm spec/vcr_cassettes/manageiq/providers/awesome_cloud/cloud_manager/refresher.yml
bundle exec rspec spec/models/manageiq/awesome_cloud/cloud_manager/refresher_spec.rb

Make sure that you have your config/secrets.yml file still present. Note that you might have to update the expected counts, as things in your environment have likely changed, but you now should have an updated VCR cassette.