Elasticsearch

Module elasticsearch

Important Capabilities

Capability	Status	Notes
Platform Instance	✅	Enabled by default

This plugin extracts the following:

• Metadata for indexes
• Column types associated with each index field

CLI based Ingestion

Install the Plugin

pip install 'acryl-datahub[elasticsearch]'

Config Details

Options

Note that a . is used to denote nested fields in the YAML recipe.

View All Configuration Options

Field	Required	Type	Description	Default
env		string	The environment that all assets produced by this connector belong to	PROD
platform_instance		string	The instance of the platform that all assets produced by this recipe belong to	None
host		string	The elastic search host URI.	localhost:9200
username		string	The username credential.	None
password		string	The password credential.	None
use_ssl		boolean	Whether to use SSL for the connection or not.	False
verify_certs		boolean	Whether to verify SSL certificates.	False
ca_certs		string	Path to a certificate authority (CA) certificate.	None
client_cert		string	Path to the file containing the private key and the certificate, or cert only if using client_key.	None
client_key		string	Path to the file containing the private key if using separate cert and key files.	None
ssl_assert_hostname		boolean	Use hostname verification if not False.	False
ssl_assert_fingerprint		string	Verify the supplied certificate fingerprint if not None.	None
url_prefix		string	There are cases where an enterprise would have multiple elastic search clusters. One way for them to manage is to have a single endpoint for all the elastic search clusters and use url_prefix for routing requests to different clusters.
ingest_index_templates		boolean	Ingests ES index templates if enabled.	False
index_pattern		AllowDenyPattern (see below for fields)	regex patterns for indexes to filter in ingestion.	{'allow': ['.*'], 'deny': ['^_.', '^ilm-history.'], 'ignoreCase': True}
index_pattern.allow		Array of string	List of regex patterns to include in ingestion	['.*']
index_pattern.deny		Array of string	List of regex patterns to exclude from ingestion.	[]
index_pattern.ignoreCase		boolean	Whether to ignore case sensitivity during pattern matching.	True
index_template_pattern		AllowDenyPattern (see below for fields)	The regex patterns for filtering index templates to ingest.	{'allow': ['.*'], 'deny': ['^_.*'], 'ignoreCase': True}
index_template_pattern.allow		Array of string	List of regex patterns to include in ingestion	['.*']
index_template_pattern.deny		Array of string	List of regex patterns to exclude from ingestion.	[]
index_template_pattern.ignoreCase		boolean	Whether to ignore case sensitivity during pattern matching.	True

Schema

The JSONSchema for this configuration is inlined below.

{
  "title": "ElasticsearchSourceConfig",
  "description": "Any source that connects to a platform should inherit this class",
  "type": "object",
  "properties": {
    "env": {
      "title": "Env",
      "description": "The environment that all assets produced by this connector belong to",
      "default": "PROD",
      "type": "string"
    },
    "platform_instance": {
      "title": "Platform Instance",
      "description": "The instance of the platform that all assets produced by this recipe belong to",
      "type": "string"
    },
    "host": {
      "title": "Host",
      "description": "The elastic search host URI.",
      "default": "localhost:9200",
      "type": "string"
    },
    "username": {
      "title": "Username",
      "description": "The username credential.",
      "type": "string"
    },
    "password": {
      "title": "Password",
      "description": "The password credential.",
      "type": "string"
    },
    "use_ssl": {
      "title": "Use Ssl",
      "description": "Whether to use SSL for the connection or not.",
      "default": false,
      "type": "boolean"
    },
    "verify_certs": {
      "title": "Verify Certs",
      "description": "Whether to verify SSL certificates.",
      "default": false,
      "type": "boolean"
    },
    "ca_certs": {
      "title": "Ca Certs",
      "description": "Path to a certificate authority (CA) certificate.",
      "type": "string"
    },
    "client_cert": {
      "title": "Client Cert",
      "description": "Path to the file containing the private key and the certificate, or cert only if using client_key.",
      "type": "string"
    },
    "client_key": {
      "title": "Client Key",
      "description": "Path to the file containing the private key if using separate cert and key files.",
      "type": "string"
    },
    "ssl_assert_hostname": {
      "title": "Ssl Assert Hostname",
      "description": "Use hostname verification if not False.",
      "default": false,
      "type": "boolean"
    },
    "ssl_assert_fingerprint": {
      "title": "Ssl Assert Fingerprint",
      "description": "Verify the supplied certificate fingerprint if not None.",
      "type": "string"
    },
    "url_prefix": {
      "title": "Url Prefix",
      "description": "There are cases where an enterprise would have multiple elastic search clusters. One way for them to manage is to have a single endpoint for all the elastic search clusters and use url_prefix for routing requests to different clusters.",
      "default": "",
      "type": "string"
    },
    "index_pattern": {
      "title": "Index Pattern",
      "description": "regex patterns for indexes to filter in ingestion.",
      "default": {
        "allow": [
          ".*"
        ],
        "deny": [
          "^_.*",
          "^ilm-history.*"
        ],
        "ignoreCase": true
      },
      "allOf": [
        {
          "$ref": "#/definitions/AllowDenyPattern"
        }
      ]
    },
    "ingest_index_templates": {
      "title": "Ingest Index Templates",
      "description": "Ingests ES index templates if enabled.",
      "default": false,
      "type": "boolean"
    },
    "index_template_pattern": {
      "title": "Index Template Pattern",
      "description": "The regex patterns for filtering index templates to ingest.",
      "default": {
        "allow": [
          ".*"
        ],
        "deny": [
          "^_.*"
        ],
        "ignoreCase": true
      },
      "allOf": [
        {
          "$ref": "#/definitions/AllowDenyPattern"
        }
      ]
    }
  },
  "additionalProperties": false,
  "definitions": {
    "AllowDenyPattern": {
      "title": "AllowDenyPattern",
      "description": "A class to store allow deny regexes",
      "type": "object",
      "properties": {
        "allow": {
          "title": "Allow",
          "description": "List of regex patterns to include in ingestion",
          "default": [
            ".*"
          ],
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "deny": {
          "title": "Deny",
          "description": "List of regex patterns to exclude from ingestion.",
          "default": [],
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "ignoreCase": {
          "title": "Ignorecase",
          "description": "Whether to ignore case sensitivity during pattern matching.",
          "default": true,
          "type": "boolean"
        }
      },
      "additionalProperties": false
    }
  }
}

Code Coordinates

• Class Name: datahub.ingestion.source.elastic_search.ElasticsearchSource
• Browse on GitHub

Questions

If you've got any questions on configuring ingestion for Elasticsearch, feel free to ping us on our Slack