External repositories

Infrahub supports two different types of connections to external Git repositories:

• Repository fully integrates with Git version control, including branch tracking and two-way branch synchronization.
• Read-only Repository links a particular branch in Infrahub to a particular ref in the Git repository. It will only read from the Git repository. It will never make any changes to the external repository.

Recommendations

• We recommend to use a dedicated repository for Infrahub resources.
• We recommend to configure the branch main as default_branch.
• Use Read-only Repository preferably for slow moving assets as pulling the changes is a manual operation. It goes well with putting a release tag as a ref and bump the ref from release to release.
• Use Read-only Repository for shared assets across multiple instances or repository external to your organization.

Adding a repository

You will need to submit an access token with your request to create a repository in Infrahub. Infrahub uses your username and this token to connect to the external Git repository.

• Via the web interface
• Via the GraphQL interface
• Via the Infrahub SDK

info

If your repository is private you might want to have a personal token to access the repository. If you use GitHub as your Git Server you can find further information in this Guide

Via the web interface

1. Log in to the Infrahub UI
2. Go to Unified Storage > Repository
3. Click on the + Add Git Repository button
4. Select the repository type
5. Complete the required information:

Repository location

The URL of the external repository, for example https://github.com/opsmill/infrahub.git.

Authentication

(Optional): Credential object holding your username / password or personal token with access to the Git repository specified.

Name

The name you want to give the repository in Infrahub for identification purposes.

Description

(Optional): A description or comment about the repository used for informational purposes.

Ref

(Optional): Branch, tag or commit reference to pull.

Tags

(Optional): Assign any tags to be associated with the repository.

Validate that everything is correct

In the UI, you should see your new repository. If the repository you added has Unknown as Operational Status it means that Infrahub didn't managed to reach your repository. Verify the location and credentials.

Via the GraphQL interface

Using the GraphQL Interface, it is possible to add a Repository or Read-only Repository via a Mutation.

1. Open the GraphQL Interface.
2. If needed, i.e., your repository is private, create a Credential object to hold your username / password.

  # Endpoint: http://127.0.0.1:8000/graphql/main
  mutation {
    CorePasswordCredentialCreate(
      data: {
        name: {value: "My Git Credential"},
        username: {value: "MY_USERNAME"},
        password: {value: "MY_TOKEN_OR_PASSWORD"}
      }
    ) {
      ok
      object {
        id
      }
    }
  }

3. Copy-paste the correct mutation from below and complete the information

Repository

  # Endpoint: http://127.0.0.1:8000/graphql/main
  mutation {
    CoreRepositoryCreate(
      data: {
        name: {value: "My Git Repository"},
        location: {value: "https://GIT_SERVER/YOUR_GIT_USERNAME/YOUR_REPOSITORY_NAME.git"},
        # credential: {id: "CREDENTIAL_ID_FROM_PREVIOUS_REQUEST"} <-- optional: This needs to be the credential id created at step 2
      }
    ) {
      ok
      object {
        id
      }
    }
  }

Read-only Repository

Make sure that you are on the correct Infrahub branch. Unlike a Repository, a Read-only Repository will only pull files into the Infrahub branch on which it was created.

  # Endpoint : http://127.0.0.1:8000/graphql/<branch>
  mutation {
    CoreReadOnlyRepositoryCreate(
      data: {
        name: {value: "My Git Repository"},
        location: {value: "https://GIT_SERVER/YOUR_GIT_USERNAME/YOUR_REPOSITORY_NAME.git"},
        ref: { value: "BRANCH/TAG/COMMIT_TO_TRACK" },
        # credential: {id: "CREDENTIAL_ID_FROM_PREVIOUS_REQUEST"} <-- optional: This needs to be the credential id created at step 2
      }
    ) {
      ok
      object {
        id
      }
    }
  }

Validate that everything is correct

The repository should be visible under Unified Storage / Repository. If the repository you added has Unknown as Operational Status it means that Infrahub didn't managed to reach your repository. Verify the location and credentials.

Via the Infrahub SDK

1. Install and setup the Infrahub SDK
2. If needed, i.e., your repository is private, create a Credential object to hold your username / password.

  # Create credential object ...
  credential = client.create(
      "CorePasswordCredential",
      name="My Git Credential",
      username="MY_USERNAME",
      password="MY_TOKEN_OR_PASSWORD",
  )

  # ... and save it!
  credential.save()

3. Create the correct repository object

Repository

  # Create repository object ...
  repository = client.create(
      "CoreRepository",
      name="My Git repository",
      location="https://GIT_SERVER/YOUR_GIT_USERNAME/YOUR_REPOSITORY_NAME.git",
      # credential=credential,  <-- optional: This needs to be the credential object created at step 2
  )

  # and save it ...
  repository.save()

Read-only Repository

  # Create repository object ...
  repository = client.create(
      "CoreRepository",
      name="My Git repository",
      location="https://GIT_SERVER/YOUR_GIT_USERNAME/YOUR_REPOSITORY_NAME.git",
      ref="BRANCH/TAG/COMMIT_TO_TRACK",
      # credential=credential,  <-- optional: This needs to be the credential object created at step 2
  )

  # and save it ...
  repository.save()

Pulling changes for read-only repository

Read-only Repository and Repository work in different ways when it comes to tracking changes on the remote repository. Please refer to the Repository Topic for further details.

warning

Unlike Repository, Infrahub does not automatically update Read-only Repository with changes from the external repository. To pull in changes from the external repository you must update the ref of the Read-only Repository to the desired value.

Via the web interface

1. Log in to the Infrahub UI
2. Go to Unified Storage > Repository
3. Click on the CoreReadOnlyRepository record
4. Click on the Edit Read-Only Repository button
5. Change the required information:

Ref

(Optional): Branch, tag or commit reference to pull.

Via the GraphQL interface

1. Open the GraphQL Interface.
2. Copy-paste the correct mutation from below and complete the information

  # Endpoint : http://127.0.0.1:8000/graphql/main
  mutation {
    CoreReadOnlyRepositoryUpdate(
      data: {
        id: "ID_OF_THE_REPOSITORY"
        ref: { value: "BRANCH/TAG/COMMIT_TO_TRACK" }
      }
    ) {
      ok
      object {
        id
      }
    }
  }

Troubleshooting repository

Various issues could affect repositories. The Repository Status gives you further details about the root cause.

Also, directly from the UI you can access the output of low level repository operations:

1. Log in to the Infrahub UI
2. Go to Unified Storage > Repository
3. Click on the CoreReadOnlyRepository record
4. Click on the Tasks tab

GitHub access token

If you are using GitHub as your Git Server and your repository is private, you need to have an access token to be able to access the repository. You can use either a classic Personal Access Token or a Fine-grained Access Token.

Generate a GitHub fine-grained personal access token

1. Go to settings > Developer Settings > Personal access tokens New GitHub token
2. Select Fine-grained tokens
3. Limit the scope of the token in Repository Access > Only Select Repositories
4. Grant the token permission:

• a. If you want to create a Repository using this token, then you will need to give it Read/Write access for the Content of the repository.
• b. If you want to create a Read-only Repository using this token, then you will only need to give it Read access for the Content of the repository.

Installing custom CA certificates

If the git server hosting your external git repository is using a server certificate that is signed by a certificate authority that is not part of trust store, then you will have to add the CA certificate using the following process.

info

In this process we will have to build a custom docker image. This docker image will have to be rebuild every time you want to use a new release.

First you will have to gather the CA certificate (and any intermediate CA certificates) in PEM format and store it on you local disk. This guide makes the assumption that you have saved the CA certificate file as mycacertificate.crt

Next we will have to create a Dockerfile. The file should be saved as Dockerfile in the same directory as the directory containing the CA certificate you want to install.

ARG INFRAHUB_VERSION=0.16.0
FROM registry.opsmill.io/opsmill/infrahub:${INFRAHUB_VERSION}

COPY mycacertificate.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates

We then have to build a Docker image from the Dockerfile. We can do this by executing this command

INFRAHUB_VERSION=0.16.0 && docker build --build-arg INFRAHUB_VERSION=$INFRAHUB_VERSION -f Dockerfile -t custom/infrahub:${INFRAHUB_VERSION}

This will build a custom docker image and tag it as custom/infrahub:0.16.0. You can change the version by changing the version we set in the INFRAHUB_VERSION shell variable and you can change the tag to you own preference.

As a last step we have to create a docker-compose.override.yml file with the following contents in the development directory of your clone of the Infrahub repository.

---
services:
  infrahub-git:
    image: custom/infrahub:0.16.0

A development environment can then be spun up with invoke demo.start command as, explained in the Installing Infrahub guide.

Disable certificate verification

You can disable certificate verification for the external Git repository by following the following process.

warning

Disabling certificate validation is a BAD practice and is STRONGLY discouraged

info

In this process we will have to build a custom docker image. This docker image will have to be rebuild every time you want to use a new release.

We will have to create a Dockerfile, that should be saved as Dockerfile on your local filesystem.

ARG INFRAHUB_VERSION=0.16.0
FROM registry.opsmill.io/opsmill/infrahub:${INFRAHUB_VERSION}

RUN git config --global http.sslVerify "false"

We then have to build a Docker image from the Dockerfile. We can do this by executing this command

INFRAHUB_VERSION=0.16.0 && docker build --build-arg INFRAHUB_VERSION=$INFRAHUB_VERSION -f Dockerfile -t custom/infrahub:${INFRAHUB_VERSION}

This will build a custom docker image and tag it as custom/infrahub:0.16.0. You can change the version by changing the version we set in the INFRAHUB_VERSION shell variable and you can change the tag to you own preference.

As a last step we have to create a docker-compose.override.yml file with the following contents in the development directory of your clone of the Infrahub repository.

---
services:
  infrahub-git:
    image: custom/infrahub:0.16.0

A development environment can then be spun up with invoke demo.start command as, explained in the Installing Infrahub guide.

Using a proxy server

In some scenarios Infrahub's git worker containers are not allowed to connect directly to the git server hosting your repository and the connection has to pass through a proxy server.

info

This way of using a proxy server for git repositories only works when using HTTP(S) connections. SSH connections are not supported.

info

In this process we will have to build a custom docker image. This docker image will have to be rebuild every time you want to use a new release.

We will have to create a Dockerfile, that should be saved as Dockerfile on your local filesystem. In the Dockerfile you will have to adapt the URL for the proxy to your environment. Replace user:password with the username and password you have to use for the proxy, if required. Replace the FQDN internal.proxy to the FQDN of the proxy and modify the port, if required.

ARG INFRAHUB_VERSION=0.16.0
FROM registry.opsmill.io/opsmill/infrahub:${INFRAHUB_VERSION}

RUN git config --global http.proxy http://user:[email protected]:8080

We then have to build a Docker image from the Dockerfile. We can do this by executing this command

INFRAHUB_VERSION=0.16.0 && docker build --build-arg INFRAHUB_VERSION=$INFRAHUB_VERSION -f Dockerfile -t custom/infrahub:${INFRAHUB_VERSION}

This will build a custom docker image and tag it as custom/infrahub:0.16.0. You can change the version by changing the version we set in the INFRAHUB_VERSION shell variable and you can change the tag to you own preference.

As a last step we have to create a docker-compose.override.yml file with the following contents in the development directory of your clone of the Infrahub repository.

---
services:
  infrahub-git:
    image: custom/infrahub:0.16.0

A development environment can then be spun up with invoke demo.start command as, explained in the Installing Infrahub guide.