Source Configuration
Dremio supports a variety of source types, and the available parameters of the config object differ among the source types. This page provides examples of the config object for each supported source type and descriptions of the config object's parameters that are available for each source type.
The source object contains the same parameters for all source types except for the parameters of the config object. Read the Source page for information about the parameters all source types have in common.
Amazon OpenSearch Service
Amazon OpenSearch Service Source config Object{
"hostname": String,
"port": Number,
"authenticationType": String [ACCESS_KEY, EC2_METADATA, NONE],
"accessKey": String,
"accessSecret": String,
"overwriteRegion": Boolean,
"regionName": String,
"scriptsEnabled": Boolean,
"showHiddenIndices": Boolean,
"showIdColumn": Boolean,
"readTimeoutMillis": Number,
"scrollTimeoutMillis": Number,
"usePainless": Boolean,
"scrollSize": Number,
"allowPushdownOnNormalizedOrAnalyzedFields": Boolean,
"warnOnRowCountMismatch": Boolean,
"encryptionValidationMode": String [
CERTIFICATE_AND_HOSTNAME_VALIDATION, CERTIFICATE_ONLY_VALIDATION, NO_VALIDATION
]
}
Amazon Redshift
Amazon Redshift Source config Object{
"config": {
"connectionString": "jdbc:redshift://testing-redshift-cluster.r57mkjay4utq.us-west-2.redshift.amazonaws.com:5439/support",
"username": "redshift-owner",
"password": "$DREMIO_EXISTING_VALUE$",
"authenticationType": "MASTER",
"propertyList": [
{
"name": "ApplicationName",
"value": "testingDashboard"
}
],
"fetchSize": 200,
"maxIdleConns": 8,
"idleTimeSec": 60,
"queryTimeoutSec": 0
}
}
Amazon Redshift config Parameters
connectionString Body String
JDBC connection string for the Redshift database.
Example: jdbc:redshift://testing-redshift-cluster.r57mkjay4utq.us-west-2.redshift.amazonaws.com:5439/support
username Body String
Username for authentication with master credentials or a secret resource URL.
Example: redshift-owner
password Body String
Password for authentication with master credentials. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.
secretResourceUrl Body String
For secret-based authentication, the Amazon Resource Name (ARN) for the secret. To keep the secret secure, Dremio returns the secretResourceUrl value as $DREMIO_EXISTING_VALUE$ in API responses.
Example: arn:aws:secretsmanager:us-west-2:123456789012㊙️my-rds-secret-VNenFy
authenticationType Body String
Type of authentication for Dremio to use to connect to the source.
ANONYMOUS: No authentication is needed.MASTER: Use credentials from a master database user or use a secret resource URL.AWS_PROFILE: Use an AWS source.
Example: MASTER
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "ApplicationName","value": "testingDashboard"}]
fetchSize Body Integer Optional
Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.
Example: 200
maxIdleConns Body Integer Optional
Maximum number of connections that can be idle at any given time. Default is 8.
Example: 8
idleTimeSec Body Integer Optional
Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.
Example: 60
queryTimeoutSec Body Integer
Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.
Example: 0
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: ApplicationName
value Body String Optional
Value for the connection property.
Example: testingDashboard
Amazon S3
Amazon S3 Source config Object{
"config": {
"accessKey": "EXAMPLE78HT89VS4YJEL",
"accessSecret": "$DREMIO_EXISTING_VALUE$",
"secure": true,
"externalBucketList": [
"external_bucket_1",
"external_bucket_2"
],
"propertyList": [
{
"name": "fs.s3a.proxy.host",
"value": "proxyHost.example.com"
}
],
"rootPath": "/",
"enableAsync": true,
"compatibilityMode": false,
"isCachingEnabled": true,
"maxCacheSpacePct": 100,
"whitelistedBuckets": [
"archive.dremio.com",
"logs_east-1",
"logs_west-1"
],
"requesterPays": false,
"enableFileStatusCheck": true,
"defaultCtasFormat": "ICEBERG",
"isPartitionInferenceEnabled": false,
"credentialType": "ACCESS_KEY"
}
}
Amazon S3 config Parameters
accessKey Body String
AWS access key. Required for credentialType ACCESS_KEY.
Example: EXAMPLE78HT89VS4YJEL
accessSecret Body String
AWS access secret. Required for credentialType ACCESS_KEY. To keep the access secret secure, Dremio returns the accessSecret value as $DREMIO_EXISTING_VALUE$ in API responses.
assumedRoleARN Body String
Amazon Resource Name (ARN) for the AWS Identity and Access Management (IAM) role to assume. Required for credentialType ACCESS_KEY and EC2_METADATA.
Example: arn:aws:iam::594632595346:role/OrganizationAccountAccessRole
awsProfile Body String
AWS profile name. If you do not provide a profile name, Dremio uses the default profile.
Example: example-profile
secure Body Boolean Optional
If set to enable a secure connection with SSL encryption between the S3 bucket and Dremio, the value is true (default). Otherwise, the value is false.
Example: true
externalBucketList Body Array of String Optional
List of external buckets that are not included with the specified AWS account credentials.
Example: ["external_bucket_1","external_bucket_2"]
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "fs.s3a.proxy.host","value": "proxyHost.example.com"}]
rootPath Body String Optional
Root path of the S3 bucket. Default is /.
Example: /
enableAsync Body Boolean Optional
To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.
Example: true
compatibilityMode Body Boolean Optional
To enable the use of S3-compatible storage, set to true. Otherwise, set to false (default).
Example: false
isCachingEnabled Body Boolean Optional
To enable local caching, set to true (default). Otherwise, set to false.
Example: true
maxCacheSpacePct Body Integer Optional
Maximum percentage of the total available cache space to use on any single executor node. Used only when isCachingEnabled is set to true. Minimum is 1. Maximum is 100 (default).
Example: 100
whitelistedBuckets Body Array of String Optional
List of approved S3 buckets to include in the source. Buckets entered must be valid. Misspelled or non-existent buckets will not appear in the resulting source. Omit to include all buckets that are available in the source. If you omit the whitelistedBuckets array in a PUT request, Dremio updates the source to include all available buckets. To keep existing approved buckets while making other updates, duplicate the existing whitelistedBuckets array in the PUT request.
Example: ["archive.dremio.com","logs_east-1","logs_west-1"]
requesterPays Body Boolean Optional
If the requester (instead of the bucket owner) pays the cost of the S3 request and the data downloaded from the S3 bucket, set to true. Otherwise, set to false (default).
Example: false
enableFileStatusCheck Body Boolean Optional
To allow Dremio to check whether a file exists in the S3 bucket before handling errors gracefully, set to true (default). Otherwise, set to false. If no files are missing from the S3 bucket or a file's access permissions have not changed, set to false to reduce communication with the S3 bucket.
Example: true
defaultCtasFormat Body String Optional
Default format for the tables you create in Dremio. Default is ICEBERG.
Enum: ICEBERG, PARQUET
Example: ICEBERG
isPartitionInferenceEnabled Body Boolean Optional
To enable partition column inference, set to true. Otherwise, set to false (default). For more information, read Partition Column Inference.
Example: false
credentialType Body String Optional
Type of credential for Dremio to use to connect to the source.
Enum: NONE, EC2_METADATA, ACCESS_KEY, AWS_PROFILE
Example: ACCESS_KEY
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: fs.s3a.proxy.host
value Body String Optional
Value for the connection property.
Example: proxyHost.example.com
AWS Glue Data Catalog
AWS Glue Data Catalog Source config Object{
"config": {
"regionNameSelection": "US_EAST_1",
"accessKey": "EXAMPLE78HT89VS4YJEL",
"accessSecret": "$DREMIO_EXISTING_VALUE$",
"secure": true,
"enableAsync": true,
"isCachingEnabled": true,
"maxCacheSpacePct": 100,
"assumedRoleARN": "arn:aws:iam::594632595346:role/OrganizationAccountAccessRole",
"lakeFormationEnableAccessPermissions": false,
"defaultCtasFormat": "ICEBERG",
"propertyList": [
{
"name": "hive.metastore.warehouse.dir",
"value": "s3a://example.com/buckets/glue_location"
}
],
"credentialType": "ACCESS_KEY"
}
}
Amazon Glue Data Catalog config Parameters
regionNameSelection Body String
AWS region where the Glue Data Catalog is located.
Example: US_EAST_1
accessKey Body String
AWS access key.
Example: EXAMPLE78HT89VS4YJEL
accessSecret Body String
AWS access secret. To keep the access secret secure, Dremio returns the accessSecret value as $DREMIO_EXISTING_VALUE$ in API responses.
awsProfile Body String
AWS profile name. If you do not provide a profile name, Dremio uses the default profile.
Example: example-profile
secure Body Boolean Optional
To enable a secure connection with SSL encryption between the Glue Data Catalog and Dremio, set to true (default). Otherwise, set to false.
Example: true
enableAsync Body Boolean Optional
To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.
Example: true
isCachingEnabled Body Boolean Optional
To enable local caching, set to true (default). Otherwise, set to false.
Example: true
maxCacheSpacePct Body Integer Optional
Maximum percentage of the total available cache space to use on any single executor node. Used only when isCachingEnabled is set to true. Minimum is 1. Maximum is 100 (default).
Example: 100
assumedRoleARN Body String
Amazon Resource Name (ARN) for the AWS Identity and Access Management (IAM) role to assume.
Example: arn:aws:iam::594632595346:role/OrganizationAccountAccessRole
lakeFormationEnableAccessPermissions Body Boolean
To enforce Lake Formation access permissions on datasets so that Dremio confirms whether users have the required permissions to perform queries, set to true. Otherwise, set to false (default).
Example: false
defaultCtasFormat Body String Optional
Default format for the tables you create in Dremio. Default is ICEBERG.
Enum: ICEBERG, PARQUET
Example: ICEBERG
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "hive.metastore.warehouse.dir","value": "s3a://example.com/buckets/glue_location"}]
credentialType Body String Optional
Type of credential for Dremio to use to connect to the source.
Enum: NONE, EC2_METADATA, ACCESS_KEY, AWS_PROFILE
Example: ACCESS_KEY
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: hive.metastore.warehouse.dir
value Body String Optional
Value for the connection property.
Example: s3a://example.com/buckets/glue_location
Azure Data Lake Storage Gen1
Azure Data Lake Storage Gen1 Source config Object{
"config": {
"accountName": "datalake1",
"clientId": "98c23728-3971-4de3-a15f-61f483f51eca",
"clientKeyRefreshUrl": "https://login.microsoftonline.com/88ba5c34-c9af-11ed-b49f-325096b39f47/oauth2/token",
"clientKeyPassword": "$DREMIO_EXISTING_VALUE$",
"rootPath": "/",
"enableAsync": true,
"isCachingEnabled": true,
"maxCacheSpacePct": 100,
"defaultCtasFormat": "ICEBERG",
"propertyList": [
{
"name": "dfs.adls.oauth2.access.token.provider.type",
"value": "ClientCredential"
}
],
"isPartitionInferenceEnabled": false
}
}
Azure Data Lake Storage Gen1 config Parameters
accountName Body String
Name for the Azure Data Lake Storage Gen1 account.
Example: datalake1
clientId Body String (UUID)
Unique identifier that Azure assigned to the application when it was registered.
Example: 98c23728-3971-4de3-a15f-61f483f51eca
clientKeyRefreshUrl Body String
Azure Active Directory OAuth 2.0 Token Endpoint for the registered application.
Example: https://login.microsoftonline.com/88ba5c34-c9af-11ed-b49f-325096b39f47/oauth2/token
clientKeyPassword Body String Optional
Password value for the registered application. To keep the password secure, Dremio returns the clientKeyPassword value as $DREMIO_EXISTING_VALUE$ in API responses.
rootPath Body String Optional
Root path of the Azure Data Lake Storage Gen1 source. Default is /.
Example: /
enableAsync Body Boolean Optional
To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.
Example: true
isCachingEnabled Body Boolean Optional
To enable local caching, set to true (default). Otherwise, set to false.
Example: true
maxCacheSpacePct Body Integer Optional
Maximum percentage of the total available cache space to use on any single executor node. Used only when isCachingEnabled is set to true. Minimum is 1. Maximum is 100 (default).
Example: 100
defaultCtasFormat Body String Optional
Default format for the tables you create in Dremio. Default is ICEBERG.
Enum: ICEBERG, PARQUET
Example: ICEBERG
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "dfs.adls.oauth2.access.token.provider.type","value": "ClientCredential"}]
isPartitionInferenceEnabled Body Boolean Optional
To enable partition column inference, set to true. Otherwise, set to false (default). For more information, read Partition Column Inference.
Example: false
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: dfs.adls.oauth2.access.token.provider.type
value Body String Optional
Value for the connection property.
Example: ClientCredential
Azure Storage
Azure Storage Source config Object{
"config": {
"accountKind": "STORAGE_V2",
"accountName": "azurestoragev2accountname",
"accessKey": "$DREMIO_EXISTING_VALUE$",
"rootPath": "/",
"enableSSL": true,
"enableAsync": true,
"credentialsType": "ACCESS_KEY",
"isCachingEnabled": true,
"maxCacheSpacePct": 100,
"defaultCtasFormat": "ICEBERG",
"propertyList": [
{
"name": "dremio.azure.mode",
"value": "STORAGE_V2"
}
],
"isPartitionInferenceEnabled": false
}
}
Azure Storage config Parameters
accountKind Body String
Type of Azure Storage account.
Enum: STORAGE_V1, STORAGE_V2
Example: STORAGE_V2
accountName Body String
Name of the Azure Storage account.
Example: azurestoragev2accountname
accessKey Body String
Azure access key. To keep the access key secure, Dremio returns the accessKey value as $DREMIO_EXISTING_VALUE$ in API responses.
clientSecret Body String
Azure client secret. To keep the client secret secure, Dremio returns the clientSecret value as $DREMIO_EXISTING_VALUE$ in API responses.
rootPath Body String Optional
Root path of the Azure Storage source. Default is /.
Example: /
enableSSL Body Boolean Optional
To enable a secure connection with SSL encryption between the Azure Storage source and Dremio, set to true (default). Otherwise, set to false.
Example: true
enableAsync Body Boolean Optional
To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.
Example: true
credentialType Body String Optional
Type of credential for Dremio to use to connect to the source.
Enum: ACCESS_KEY, AZURE_ACTIVE_DIRECTORY
Example: ACCESS_KEY
isCachingEnabled Body Boolean Optional
To enable local caching, set to true (default). Otherwise, set to false.
Example: true
maxCacheSpacePct Body Integer Optional
Maximum percentage of the total available cache space to use on any single executor node. Used only when isCachingEnabled is set to true. Minimum is 1. Maximum is 100 (default).
Example: 100
defaultCtasFormat Body String Optional
Default format for the tables you create in Dremio. Default is ICEBERG.
Enum: ICEBERG, PARQUET
Example: ICEBERG
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "dremio.azure.mode","value": "STORAGE_V2"}]
isPartitionInferenceEnabled Body Boolean Optional
To enable partition column inference, set to true. Otherwise, set to false (default). For more information, read Partition Column Inference.
Example: false
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: dremio.azure.mode
value Body String Optional
Value for the connection property.
Example: STORAGE_V2
Dremio-to-Dremio Connector
Dremio-to-Dremio Connector Source config Object{
"config": {
"hostType": "DIRECT",
"hostname": "35.219.202.153",
"port": "31010",
"username": "exampleuser",
"password": "$DREMIO_EXISTING_VALUE$",
"useSsl": false,
"userImpersonation": false,
"maxIdleConns": 8,
"idleTimeSec": 60,
"queryTimeoutSec": 0,
"propertyList": [
{
"name": "disableCertificateVerification",
"value": "false"
}
]
}
}
Dremio-to-Dremio Connector config Parameters
hostType Body String
Connection method to use. To connect directly to a coordinator node of the cluster, DIRECT. To connect to an external ZooKeeper instance that coordinates the nodes of the cluster, ZOOKEEPER.
Enum: DIRECT, ZOOKEEPER
Example: DIRECT
hostname Body String
Hostname of the coordinator node or ZooKeeper instance.
Example: 35.219.202.153
port Body String
Port number of the coordinator node or ZooKeeper instance. Default is 31010.
Example: 31010
username Body String
Username for authentication.
Example: exampleuser
password Body String
Password for authentication. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.
useSsl Body Boolean Optional
If the source cluster is configured to use TLS for connections, set to true. Otherwise, set to false.
Example: false
userImpersonation Body Boolean Optional
To allow users to run queries on the source cluster under their own user IDs rather than the user ID for the account used to authenticate, set to true. Otherwise, set to false (default). If set to true, the source cluster must be configured to allow inbound impersonation. For more information, read Inbound Impersonation.
Example: false
maxIdleConns Body Integer Optional
Maximum number of connections that can be idle at any given time. Default is 8.
Example: 8
idleTimeSec Body Integer Optional
Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.
Example: 60
queryTimeoutSec Body Integer Optional
Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.
Example: 0
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "disableCertificateVerification","value": "false"}]
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: disableCertificateVerification
value Body String Optional
Value for the connection property.
Example: false
Elasticsearch
Elasticsearch Source config Object{
"config": {
"scriptsEnabled": true,
"showHiddenIndices": false,
"showIdColumn": false,
"readTimeoutMillis": 60000,
"scrollTimeoutMillis": 300000,
"usePainless": true,
"scrollSize": 4000,
"allowPushdownOnNormalizedOrAnalyzedFields": false,
"warnOnRowCountMismatch": false,
"encryptionValidationMode": "CERTIFICATE_AND_HOSTNAME_VALIDATION",
"forceDoublePrecision": false,
"hostList": [
{
"hostname": "172.25.0.208",
"port": 9200
}
],
"authenticationType": "ANONYMOUS",
"sslEnabled": false,
"useWhitelist": false
}
}
Elasticsearch config Parameters
scriptsEnabled Body Boolean Optional
If Dremio should use script pushdowns, set to true (default). Otherwise, set to false.
Example: true
showHiddenIndices Body Boolean Optional
To show hidden indices, set to true. Otherwise, set to false (default).
Example: false
showIdColumn Body Boolean Optional
To show the elasticsearch ID column, set to true. Otherwise, set to false (default).
Example: false
readTimeoutMillis Body Integer Optional
Time to wait to read data from the source, in milliseconds. Default is 60000.
Example: 60000
scrollTimeoutMillis Body Integer Optional
Time to wait for each scroll request, in milliseconds. Default is 300000.
Example: 300000
usePainless Body Boolean Optional
To use the Painless scripting language when connecting to Elasticsearch 5.0+, set to true (default). Otherwise, set to false.
Example: true
scrollSize Body Integer Optional
Scroll size for Dremio's Elasticsearch requests. Default is 4000. The scrollSize value must be less than or equal to the setting for index.max_result_window in Elasticsearch.
Example: 4000
allowPushdownOnNormalizedOrAnalyzedFields Body Boolean Optional
To enable pushdown filters and aggregations on analyzed text fields and normalized keyword fields, set to true. Otherwise, set to false (default). May produce unexpected results when enabled.
Example: false
warnOnRowCountMismatch Body Boolean Optional
If Dremio should warn the user when a query returns fewer Elasticsearch records than expected instead of failing the query, set to true. Otherwise, set to false (default).
Example: false
encryptionValidationMode Body String Optional
Method to use to validate data encryption for the source.
Enum: CERTIFICATE_AND_HOSTNAME_VALIDATION, CERTIFICATE_ONLY_VALIDATION, NO_VALIDATION
Example: CERTIFICATE_AND_HOSTNAME_VALIDATION
forceDoublePrecision Body Boolean Optional
To force precision for double values, set to true. Otherwise, set to false (default).
Example: false
hostList Body Array of Object
Information about Elasticsearch hosts. Each object in the hostList includes the hostname and the corresponding port for the host.
Example: [{"hostname": "172.25.0.208","port": 9200}]
authenticationType Body String
Type of authentication for Dremio to use to connect to the source.
Enum: ANONYMOUS, MASTER
Example: ANONYMOUS
username Body String
Username for authenticating with master credentials.
Example: exampleuser
password Body String
Password for authenticating with master credentials. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.
sslEnabled Body Boolean Optional
To enable a secure connection with SSL encryption between Elasticsearch and Dremio, set to true. Otherwise, set to false (default).
Example: false
useWhitelist Body Boolean Optional
To query only the hosts specified in the hostList, set to true. Otherwise, set to false (default).
Example: false
Parameters of the hostList Object
hostname Body String Optional
Name of the host to use to connect to the Elasticsearch source.
Example: 172.25.0.208
port Body Integer Optional
Port to use with the specified hostname to connect to the Elasticsearch source. Default is 9200.
Example: 9200
Google Cloud Storage
Google Cloud Storage Source config Object{
"config": {
"projectId": "dremio-4321",
"authMode": "AUTO",
"rootPath": "/",
"asyncEnabled": true,
"cachingEnable": true,
"cachePercent": 70,
"privateKeyId": "",
"clientEmail": "",
"clientId": "",
"defaultCtasFormat": "ICEBERG",
"propertyList": [
{
"name": "dremio.gcs.clientEmail",
"value": "exampleuser@dremio-project.iam.gserviceaccount.com"
}
],
"isPartitionInferenceEnabled": false
}
}
Google Cloud Storage config Parameters
projectId Body String
Project ID for the Google Cloud Storage project.
Example: dremio-4321
authMode Body String
Type of authentication for Dremio to use to connect to the source.
Enum: AUTO, SERVICE_ACCOUNT_KEYS"
Example: AUTO
rootPath Body String Optional
Root path of the Google Cloud Storage source. Default is /.
Example: /
asyncEnabled Body Boolean Optional
To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.
Example: true
cachingEnable Body Boolean Optional
To enable local caching, set to true (default). Otherwise, set to false.
Example: true
cachePercent Body Integer Optional
Maximum percentage of the total available cache space to use on any single executor node. Used only when cachingEnable is set to true. Default is 70. Minimum is 1. Maximum is 100.
Example: 70
privateKeyId Body String
Service account key ID for the Google Cloud Storage service account. Required only if authenticating with the Service Account Keys method.
Example: f302b86d0247be19393bb66ea142887fc9621360
privateKey Body String
Service account key for the Google Cloud Storage service account. Required only if authenticating with the Service Account Keys method. To keep the service account key secure, Dremio returns the privateKey value as $DREMIO_EXISTING_VALUE$ in API responses.
clientEmail Body String
Email address associated with the Google Cloud Storage service account. Required only if authenticating with the Service Account Keys method.
Example: exampleuser@dremio-project.iam.gserviceaccount.com
clientId Body String
Client ID for the key pair. Required only if authenticating with the Service Account Keys method.
Example: 243769857672272684657
defaultCtasFormat Body String Optional
Default format for the tables you create in Dremio. Default is ICEBERG.
Enum: ICEBERG, PARQUET
Example: ICEBERG
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "dremio.gcs.clientEmail","value": "exampleuser@dremio-project.iam.gserviceaccount.com"}]
isPartitionInferenceEnabled Body Boolean Optional
To enable partition column inference, set to true. Otherwise, set to false (default). For more information, read Partition Column Inference.
Example: false
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: dremio.gcs.clientEmail
value Body String Optional
Value for the connection property.
Example: exampleuser@dremio-project.iam.gserviceaccount.com
Hadoop Distributed File System (HDFS)
HDFS Source config Object{
"config": {
"hostname": "172.23.0.208",
"port": 8020,
"enableImpersonation": false,
"rootPath": "/",
"shortCircuitFlag": "SYSTEM",
"enableAsync": true,
"isCachingEnabled": false,
"maxCacheSpacePct": 100,
"defaultCtasFormat": "ICEBERG",
"isPartitionInferenceEnabled": false,
"impersonationUserDelegationMode": "AS_IS",
"propertyList": [
{
"name": "dfs.client.socket-timeout",
"value": "120000"
}
],
"vdsAccessDelegationEnabled": true
}
}
HDFS config Parameters
hostname Body String
HDFS NameNode hostname to use to connect to the HDFS source.
Example: 172.23.0.208
port Body Integer
HDFS NameNode port to use to connect to the HDFS source. Default is 8020.
Example: 8020
enableImpersonation Body Boolean Optional
To enable user-specific file access permissions by turning on impersonation in HDFS sources, set to true. Otherwise, set to false (default).
Example: false
rootPath Body String Optional
Root path of the HDFS source. Default is /.
Example: /
shortCircuitFlag Body String Optional
Implementation status for short-circuit local reads. Default is SYSTEM.
Enum: ENABLED, DISABLED, SYSTEM
Example: SYSTEM
shortCircuitSocketPath Body String Optional
Socket path to use, if shortCircuitFlag is enabled.
Example: /var/lib/hadoop-hdfs/dn-socket
enableAsync Body Boolean Optional
To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.
Example: true
isCachingEnabled Body Boolean Optional
To enable local caching, set to true. Otherwise, set to false (default).
Example: false
maxCacheSpacePct Body Integer Optional
Maximum percentage of the total available cache space to use on any single executor node. Used only when cachingEnable is set to true. Minimum is 1. Maximum is 100 (default).
Example: 100
defaultCtasFormat Body String Optional
Default format for the tables you create in Dremio. Default is ICEBERG.
Enum: ICEBERG, PARQUET
Example: ICEBERG
isPartitionInferenceEnabled Body Boolean Optional
To enable partition column inference, set to true. Otherwise, set to false (default). For more information, read Partition Column Inference.
Example: false
impersonationUserDelegationMode Body String Optional
Capitalization method to use for impersonation usernames. Default is AS_IS.
Enum: AS_IS, LOWERCASE, UPPERCASE
Example: AS_IS
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "dfs.client.socket-timeout","value": "120000"}]
vdsAccessDelegationEnabled Body Boolean Optional
To use the view owner as the impersonated username when enableImpersonation is enabled, set to true (default). To use the user who submits the query as the impersonated username, set to false.
Example: true
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: dfs.client.socket-timeout
value Body String Optional
Value for the connection property.
Example: 120000
Hive 2.x
Hive 2.x Source config Object{
"config": {
"hostname": "172.23.0.208",
"port": 9083,
"enableSasl": false,
"propertyList": [
{
"name": "hive.server2.enable.doAs",
"value": "false"
}
],
"secretPropertyList": [
{
"name": "fs.s3a.secret.key",
"value": "70SPup32UsIZaA6c2n6bf3rQONTD6Zn6OqvcUhBy"
}
],
"enableAsync": true,
"isCachingEnabledForS3AndAzureStorage": true,
"isCachingEnabledForHDFS": true,
"maxCacheSpacePct": 100,
"defaultCtasFormat": "ICEBERG",
"authType": "STORAGE",
"rangerServiceName": "",
"rangerHostURL": "",
"impersonationUserDelegationMode": "AS_IS",
"vdsAccessDelegationEnabled": true
}
}
Hive 2.x config Parameters
hostname Body String
IP address for the Hive metastore host.
Example: 172.23.0.208
port Body Integer
Port to use with the specified hostname to connect to the Hive source. Default is 9083.
Example: 9083
enableSasl Body Boolean Optional
To enable Simple Authentication and Security Layer (SASL), set to true (default). Otherwise, set to false.
Example: false
kerberosPrincipal Body String Optional
Name of the Kerberos principal identity for SASL. Required only if enableSasl is true.
Example: primary/instance@REALM
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "hive.server2.enable.doAs","value": "false"}]
secretPropertyList Body Array of Object Optional
Additional connection properties for the source. The values are kept masked in the Credentials section of the Advanced Options page of the Source Settings dialog in the Dremio UI. Each object includes the name of the property and the corresponding value to use and to keep secret.
Example: [{"name": "fs.s3a.secret.key","value": "70SPup32UsIZaA6c2n6bf3rQONTD6Zn6OqvcUhBy"}]
enableAsync Body Boolean Optional
To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.
Example: true
isCachingEnabledForS3AndAzureStorage Body Boolean Optional
To enable local caching for Amazon S3 and Azure Storage, set to true (default). Otherwise, set to false.
Example: true
isCachingEnabledForHDFS Body Boolean Optional
To enable local caching for Hadoop Distributed File System (HDFS), set to true. Otherwise, set to false (default).
Example: false
maxCacheSpacePct Body Integer Optional
Maximum percentage of the total available cache space to use on any single executor node. Minimum is 1. Maximum is 100 (default).
Example: 100
defaultCtasFormat Body String Optional
Default format for the tables you create in Dremio. Default is ICEBERG.
Enum: ICEBERG, PARQUET
Example: ICEBERG
authType Body Boolean Optional
Type of authentication for Dremio to use to connect to the source. Default is STORAGE.
Enum: STORAGE, SQL, RANGER
Example: STORAGE
rangerServiceName Body String Optional
The Ranger service name, if you are using Ranger-based authentication.
Example: hivedev
rangerHostURL Body String Optional
The Ranger host URL, if you are using Ranger-based authentication.
Example: http://example.com/6080
impersonationUserDelegationMode Body String Optional
Capitalization method to use for impersonation usernames. Default is AS_IS.
Enum: AS_IS, LOWERCASE, UPPERCASE
Example: AS_IS
vdsAccessDelegationEnabled Body Boolean Optional
To use the view owner as the impersonated username when enableImpersonation is enabled, set to true (default). To use the user who submits the query as the impersonated username, set to false.
Example: true
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: hive.server2.enable.doAs
value Body String Optional
Value for the connection property.
Example: false
Parameters of the secretPropertyList Object
name Body String Optional
Name of the connection property.
Example: fs.s3a.secret.key
value Body String Optional
Value for the connection property. The value is masked in the Dremio UI.
Example: 70SPup32UsIZaA6c2n6bf3rQONTD6Zn6OqvcUhBy
Hive 3.x
Hive 3.x Source config Object{
"config": {
"hostname": "172.23.0.177",
"port": 9084,
"enableSasl": false,
"propertyList": [
{
"name": "fs.s3a.aws.credentials.provider",
"value": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"
}
],
"secretPropertyList": [
{
"name": "fs.s3a.secret.key",
"value": "70SPup32UsIZaA6c2n6bf3rQONTD6Zn6OqvcUhBy"
}
],
"enableAsync": true,
"isCachingEnabledForS3AndAzureStorage": true,
"isCachingEnabledForHDFS": false,
"maxCacheSpacePct": 100,
"defaultCtasFormat": "ICEBERG",
"authType": "STORAGE",
"rangerServiceName": "",
"rangerHostURL": "",
"impersonationUserDelegationMode": "AS_IS",
"vdsAccessDelegationEnabled": true
}
}
Hive 3.x config Parameters
hostname Body String
IP address for the Hive metastore host.
Example: 172.23.0.177
port Body Integer
Port to use with the specified hostname to connect to the Hive source. Default is 9083.
Example: 9084
enableSasl Body Boolean Optional
To enable Simple Authentication and Security Layer (SASL), set to true (default). Otherwise, set to false.
Example: false
kerberosPrincipal Body String Optional
Name of the Kerberos principal identity for SASL. Required only if enableSasl is true.
Example: primary/instance@REALM
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "fs.s3a.aws.credentials.provider","value": "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"}]
secretPropertyList Body Array of Object Optional
Additional connection properties for the source. The values are kept masked in the Credentials section of the Advanced Options page of the Source Settings dialog in the Dremio UI. Each object includes the name of the property and the corresponding value to use and to keep secret.
Example: [{"name": "fs.s3a.secret.key","value": "70SPup32UsIZaA6c2n6bf3rQONTD6Zn6OqvcUhBy"}]
enableAsync Body Boolean Optional
To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.
Example: true
isCachingEnabledForS3AndAzureStorage Body Boolean Optional
To enable local caching for Amazon S3 and Azure Storage, set to true (default). Otherwise, set to false.
Example: true
isCachingEnabledForHDFS Body Boolean Optional
To enable local caching for Hadoop Distributed File System (HDFS), set to true. Otherwise, set to false (default).
Example: false
maxCacheSpacePct Body Integer Optional
Maximum percentage of the total available cache space to use on any single executor node. Minimum is 1. Maximum is 100 (default).
Example: 100
defaultCtasFormat Body String Optional
Default format for the tables you create in Dremio. Default is ICEBERG.
Enum: ICEBERG, PARQUET
Example: ICEBERG
authType Body Boolean Optional
Type of authentication for Dremio to use to connect to the source. Default is STORAGE.
Enum: STORAGE, SQL, RANGER
Example: STORAGE
rangerServiceName Body String Optional
The Ranger service name, if you are using Ranger-based authentication.
Example: hivedev
rangerHostURL Body String Optional
The Ranger host URL, if you are using Ranger-based authentication.
Example: http://example.com/6080
impersonationUserDelegationMode Body String Optional
Capitalization method to use for impersonation usernames. Default is AS_IS.
Enum: AS_IS, LOWERCASE, UPPERCASE
Example: AS_IS
vdsAccessDelegationEnabled Body Boolean Optional
To use the view owner as the impersonated username when enableImpersonation is enabled, set to true (default). To use the user who submits the query as the impersonated username, set to false.
Example: true
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: fs.s3a.aws.credentials.provider
value Body String Optional
Value for the connection property.
Example: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
Parameters of the secretPropertyList Object
name Body String Optional
Name of the connection property.
Example: fs.s3a.secret.key
value Body String Optional
Value for the connection property. The value is masked in the Dremio UI.
Example: 70SPup32UsIZaA6c2n6bf3rQONTD6Zn6OqvcUhBy
IBM Db2
IBM Db2 Source config Object{
"config": {
"database": "tpch",
"hostname": "172.25.1.245",
"username": "exampleuser",
"password": "$DREMIO_EXISTING_VALUE$",
"port": "50000",
"fetchSize": 200,
"maxIdleConns": 8,
"idleTimeSec": 60,
"queryTimeoutSec": 0,
"propertyList": [
{
"name": "deferPrepares",
"value": "true"
}
]
}
}
IBM Db2 config Parameters
database Body String
IBM Db2 database for Dremio to use.
Example: tpch
hostname Body String
IBM Db2 hostname.
Example: 172.25.1.245
username Body String
Username for authenticating with master credentials.
Example: exampleuser
password Body String
Password for authenticating with master credentials. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.
port Body String
IBM Db2 port number. Default is 50000.
Example: 50000
fetchSize Body Integer Optional
Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.
Example: 200
maxIdleConns Body Integer Optional
Maximum number of connections that can be idle at any given time. Default is 8.
Example: 8
idleTimeSec Body Integer Optional
Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.
Example: 60
queryTimeoutSec Body Integer Optional
Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.
Example: 0
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "deferPrepares","value": "true"}]
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: deferPrepares
value Body String Optional
Value for the connection property.
Example: true
MapR File System (MapR FS)
MapR FS Source config Object{
"config": {
"clusterName": "my.cluster.com",
"enableImpersonation": false,
"secure": false,
"rootPath": "/",
"defaultCtasFormat": "ICEBERG",
"isPartitionInferenceEnabled": false,
"impersonationUserDelegationMode": "AS_IS",
"vdsAccessDelegationEnabled": true
}
}
MapR FS config Parameters
clusterName Body String
Name of the MapR FS cluster.
Example: my.cluster.com
enableImpersonation Body Boolean Optional
To enable user-specific file access permissions by turning on impersonation in MapR FS sources, set to true. Otherwise, set to false (default).
Example: false
secure Body Boolean Optional
To enable a secure connection with SSL encryption between MapR FS and Dremio, set to true. Otherwise, set to false (default).
Example: false
rootPath Body String Optional
Root path of the MapR FS source. Default is /.
Example: /
defaultCtasFormat Body String Optional
Default format for the tables you create in Dremio. Default is ICEBERG.
Enum: ICEBERG, PARQUET
Example: ICEBERG
isPartitionInferenceEnabled Body Boolean Optional
To enable partition column inference, set to true. Otherwise, set to false (default). For more information, read Partition Column Inference.
Example: false
impersonationUserDelegationMode Body String Optional
Capitalization method to use for impersonation usernames. Default is AS_IS.
Enum: AS_IS, LOWERCASE, UPPERCASE
Example: AS_IS
vdsAccessDelegationEnabled Body Boolean Optional
To use the view owner as the impersonated username when enableImpersonation is enabled, set to true (default). To use the user who submits the query as the impersonated username, set to false.
Example: true
Microsoft Azure Data Explorer
Microsoft Azure Data Explorer Source config Object{
"config": {
"clusterUri": "https://prod.eastus.kusto.windows.net",
"tenantId": "6e334262-b0c6-4c36-9faf-380e97e42964",
"appId": "c3e4554f-514a-4912-8b75-93800f0d6c71",
"appSecret": "$DREMIO_EXISTING_VALUE$",
"databaseName": "datatypes",
"maxIdleConns": 8,
"idleTimeSec": 60,
"queryTimeoutSec": 0
}
}
Microsoft Azure Date Explorer config Parameters
clusterUri Body String
URI for the Azure Active Directory (AD) tenant that Dremio should connect to.
Example: https://prod.eastus.kusto.windows.net
tenantId Body String (UUID)
Unique identifier of the Azure AD tenant.
Example: 6e334262-b0c6-4c36-9faf-380e97e42964
appId Body String (UUID)
Unique identifier of the Azure AD application Dremio should use when connecting.
Example: c3e4554f-514a-4912-8b75-93800f0d6c71
appSecret Body String
Secret for the Azure AD application Dremio should use when connecting. To keep the secret secure, Dremio returns the appSecret value as $DREMIO_EXISTING_VALUE$ in API responses.
databaseName Body String
Default Azure Data Explorer database for Dremio to use.
Example: datatypes
maxIdleConns Body Integer Optional
Maximum number of connections that can be idle at any given time. Default is 8.
Example: 8
idleTimeSec Body Integer Optional
Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.
Example: 60
queryTimeoutSec Body Integer Optional
Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.
Example: 0
Microsoft Azure Synapse Analytics
Microsoft Azure Synapse Analytics Source config Object{
"config": {
"hostname": "synapsedata.sql.azuresynapse.net",
"port": "1433",
"username": "exampleuser",
"password": "$DREMIO_EXISTING_VALUE$",
"authenticationType": "MASTER",
"fetchSize": 200,
"useSsl": false,
"enableServerVerification": true,
"maxIdleConns": 8,
"idleTimeSec": 60,
"queryTimeoutSec": 0,
"database": "dedicatedpool",
"propertyList": [
{
"name": "applicationIntent",
"value": "ReadWrite"
}
]
}
}
Microsoft Azure Synapse Analytics config Parameters
hostname Body String
Microsoft Azure Synapse Analytics hostname.
Example: synapsedata.sql.azuresynapse.net
port Body String Optional
Microsoft Azure Synapse Analytics port number. Default is 1433.
Example: 1433
username Body String
Username for authenticating with master credentials.
Example: exampleuser
password Body String
Password for authenticating with master credentials. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.
authenticationType Body String
Type of authentication to use to allow Dremio to connect to the Microsoft Azure Synapse Analytics source.
ANONYMOUS: No authentication is needed.MASTER: Use credentials from a master database user.
Example: MASTER
fetchSize Body Integer Optional
Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.
Example: 200
useSsl Body Boolean Optional
If the Microsoft Azure Synapse Analytics source is configured to use TLS for connections, set to true. Otherwise, set to false (default).
Example: false
enableServerVerification Body Boolean Optional
To verify the certificate for the Microsoft Azure Synapse Analytics source when connecting, set to true. Otherwise, set to false (default).
Example: false
maxIdleConns Body Integer Optional
Maximum number of connections that can be idle at any given time. Default is 8.
Example: 8
idleTimeSec Body Integer Optional
Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.
Example: 60
queryTimeoutSec Body Integer Optional
Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.
Example: 0
database Body String Optional
Default Microsoft Azure Synapse Analytics database for Dremio to use.
Example: dedicatedpool
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "applicationIntent","value": "ReadWrite"}]
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: applicationIntent
value Body String Optional
Value for the connection property.
Example: ReadWrite
Microsoft SQL Server
Microsoft SQL Server Source config Object{
"config": {
"hostname": "172.25.0.10",
"port": "1433",
"username": "tpch",
"password": "$DREMIO_EXISTING_VALUE$",
"authenticationType": "MASTER",
"fetchSize": 200,
"useSsl": false,
"enableServerVerification": true,
"maxIdleConns": 8,
"idleTimeSec": 60,
"queryTimeoutSec": 0,
"showOnlyConnectionDatabase": false,
"propertyList": [
{
"name": "applicationIntent",
"value": "ReadWrite"
}
]
}
}
Microsoft SQL Server config Parameters
hostname Body String
Microsoft SQL Server hostname.
Example: 172.25.0.10
port Body String
Microsoft SQL Server port number. Default is 5432.
Example: 1433
username Body String
Username for authenticating with master credentials.
Example: tpch
password Body String
Password for authenticating with master credentials. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.
authenticationType Body String
Type of authentication to use to allow Dremio to connect to the Microsoft SQL Server source.
ANONYMOUS: No authentication is needed.MASTER: Use credentials from a master database user.
Example: MASTER
fetchSize Body Integer Optional
Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.
Example: 200
useSsl Body Boolean Optional
If the Microsoft SQL Server source is configured to use TLS for connections, set to true. Otherwise, set to false (default).
Example: false
enableServerVerification Body Boolean Optional
To verify the certificate for the Microsoft SQL Server source when connecting, set to true. Otherwise, set to false (default).
Example: false
maxIdleConns Body Integer Optional
Maximum number of connections that can be idle at any given time. Default is 8.
Example: 8
idleTimeSec Body Integer Optional
Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.
Example: 60
queryTimeoutSec Body Integer Optional
Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.
Example: 0
showOnlyConnectionDatabase Body Boolean Optional
To hide all databases that the credentialed user can access except the specified connection database, set to true. Otherwise, set to false (default).
Example: false
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "applicationIntent","value": "ReadWrite"}]
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: applicationIntent
value Body String Optional
Value for the connection property.
Example: ReadWrite
MongoDB
MongoDB Source config Object{
"config": {
"hostList": [
{
"hostname": "172.23.0.208",
"port": 27017
}
],
"useSsl": false,
"authenticationType": "ANONYMOUS",
"authDatabase": "admin",
"authenticationTimeoutMillis": 2000,
"secondaryReadsOnly": false,
"subpartitionSize": 0,
"sampleSize": 4095,
"sampleMethod": "FIRST",
"propertyList": [
{
"name": "maxPoolSize",
"value": "100"
}
],
"useCaseInsensitiveFieldNames": false
}
}
MongoDB config Parameters
hostList Body Array of Object
Information about MongoDB hosts. Each object in the hostList includes the hostname and the corresponding port for the host.
Example: [{"hostname": "172.23.0.208","port": 27017}]
useSsl Body Boolean Optional
To force a secure connection with SSL encryption between MongoDB and Dremio, set to true. Otherwise, set to false (default).
Example: false
authenticationType Body String
Type of authentication for Dremio to use to connect to the source.
Enum: ANONYMOUS, MASTER
Example: ANONYMOUS
username Body String
Username for authenticating with master credentials.
Example: exampleuser
password Body String
Password for authenticating with master credentials. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.
authDatabase Body String
Name of the MongoDB database to authenticate against.
Example: admin
authenticationTimeoutMillis Body Integer Optional
Time to wait for authentication requests, in milliseconds. Default is 2000.
Example: 2000
secondaryReadsOnly Body Boolean Optional
To disable reading from primaries, set to true. Otherwise, set to false (default). Enabling secondaryReadsOnly may degrade performance.
Example: false
subpartitionSize Body String Optional
Number of records to read by query fragments. Use subpartitionSize to increase query parallelism. Default is 0.
Example: 0
sampleSize Body Integer Optional
Number of records to read when sampling to determine the schema for a collection. Default is 4095. If the value is 0, the sample size is unlimited.
Example: 4095
sampleMethod Body String Optional
Method by which records should be read when sampling a collection to determine the schema. Default is FIRST.
Enum: FIRST, LAST
Example: FIRST
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "maxPoolSize","value": "100"}]
useCaseInsensitiveFieldNames Body Boolean Optional
To treat field names as case-insensitive, set to true. Otherwise, set to false (default). If set to true, Dremio records all known variations of a field name when learning the schema and uses them when pushing an operation down to MongoDB.
Example: false
Parameters of the hostList Object
hostname Body String
Name of the host to use to connect to the MongoDB source. If MongoDB is sharded, use the mongos host. Otherwise, use the mongod host.
Example: 172.23.0.208
port Body Integer
Port to use with the specified hostname to connect to the MongoDB source. Default is 27017.
Example: 27017
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: maxPoolSize
value Body String Optional
Value for the connection property.
Example: 100
MySQL
MySQL Source config Object{
"config": {
"hostname": "rdbms-linux-20231031.c.company-4321.internal",
"port": "3306",
"username": "exampleuser",
"password": "$DREMIO_EXISTING_VALUE$",
"authenticationType": "MASTER",
"fetchSize": 200,
"netWriteTimeout": 60,
"maxIdleConns": 8,
"idleTimeSec": 60,
"propertyList": [
{
"name": "DataSource",
"value": "sales2021"
}
],
"queryTimeoutSec": 0
}
}
mySQL config Parameters
hostname Body String
Name of the host to use to connect to the mySQL source.
Example: rdbms-linux-20221031.c.dremio-4321.internal
port Body Integer
Port to use with the specified hostname to connect to the mySQL source. Default is 3306.
Example: 3306
username Body String
Username for authentication with master credentials.
Example: exampleuser
password Body String
Password for authentication with master credentials. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.
authenticationType Body String
Type of authentication for Dremio to use to connect to the source.
Enum: ANONYMOUS, MASTER
Example: MASTER
fetchSize Body Integer Optional
Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.
Example: 200
netWriteTimeout Body Integer Optional
Length of time, in seconds, before an idle connection is eligible to be closed. Default is 60.
Example: 60
maxIdleConns Body Integer Optional
Maximum number of connections that can be idle at any given time. Default is 8.
Example: 8
idleTimeSec Body Integer Optional
Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.
Example: 60
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "DataSource","value": "sales2021"}]
queryTimeoutSec Body Integer Optional
Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.
Example: 0
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: DataSource
value Body String Optional
Value for the connection property.
Example: sales2021
Network-attached Storage (NAS)
NAS Source config Object{
"config": {
"path": "/home/dremio/",
"defaultCtasFormat": "ICEBERG",
"isPartitionInferenceEnabled": false
}
}
NAS config Parameters
path Body String
Path on the filesystem to use as the root for the source. The path must be accessible on all nodes.
Example: /home/dremio/
defaultCtasFormat Body String Optional
Default format for the tables you create in Dremio. Default is ICEBERG.
Enum: ICEBERG, PARQUET
Example: ICEBERG
isPartitionInferenceEnabled Body Boolean Optional
To enable partition column inference, set to true. Otherwise, set to false (default). For more information, read Partition Column Inference.
Example: false
Nessie
Nessie Source config Object{
"config": {
"nessieEndpoint": "http://172.25.0.138:19120/api/v2",
"nessieAuthType": "BEARER",
"nessieAccessToken": "$DREMIO_EXISTING_VALUE$",
"asyncEnabled": true,
"isCachingEnabled": true,
"maxCacheSpacePct": 100,
"credentialType": "ACCESS_KEY",
"awsAccessKey": "AKIAQ3XZRGQRKEXAMPLE",
"awsAccessSecret": "$DREMIO_EXISTING_VALUE$",
"awsRootPath": "/tmp.dremio.com/myFolder",
"propertyList": [
{
"name": "fs.s3a.proxy.host",
"value": "proxyHost.example.com"
}
],
"secure": true
}
}
Nessie config Parameters
nessieEndpoint Body String
IP address and port that you have set up for your Nessie server.
Example: http://172.25.0.138:19120/api/v2
nessieAuthType Body String Optional
Type of Nessie authentication for Dremio to use to connect to the source. If set to NONE, Nessie authentication is not enforced on the Nessie server and other Dremio users can view the source without authenticating. If set to BEARER, Nessie authentication is enforced with an OpenID bearer token (default).
Enum: NONE, BEARER
Example: BEARER
nessieAccessToken Body String
If you are using BEARER Nessie authentication, the OpenID bearer token. Omit if nessieAuthType is NONE. To keep the token secure, Dremio returns the nessieAccessToken value as $DREMIO_EXISTING_VALUE$ in API responses.
asyncEnabled Body Boolean Optional
To enable asynchronous access for the source and allow cloud caching to support simultaneous actions such as adding and editing, set to true (default). Otherwise, set to false.
Example: true
isCachingEnabled Body Boolean Optional
To enable local caching, set to true (default). Otherwise, set to false.
Example: true
maxCacheSpacePct Body Integer Optional
Maximum percentage of the total available cache space to use on any single executor node. Used only when isCachingEnabled is set to true. Minimum is 1. Maximum is 100 (default).
Example: 100
credentialType Body String Optional
Type of credential for Dremio to use to connect to S3 for storage of metadata and data files for tables and views. Default is ACCESS_KEY.
Enum: NONE, EC2_METADATA, ACCESS_KEY, AWS_PROFILE
Example: ACCESS_KEY
awsAccessKey Body String
AWS access key. Required for credentialType ACCESS_KEY.
Example: AKIAQ3XZRGQRKEXAMPLE
awsAccessSecret Body String
AWS access secret. Required for credentialType ACCESS_KEY. To keep the access secret secure, Dremio returns the accessSecret value as $DREMIO_EXISTING_VALUE$ in API responses.
awsRootPath Body String
Root path of the S3 bucket.
Example: /tmp.dremio.com/myFolder
assumedRoleARN Body String
Amazon Resource Name (ARN) for the AWS Identity and Access Management (IAM) role to assume. Required for credentialType ACCESS_KEY and EC2_METADATA.
Example: arn:aws:iam::594632595346:role/OrganizationAccountAccessRole
awsProfile Body String Optional
If you are using the credentialType AWS_PROFILE, the AWS profile name. If you do not provide a profile name, Dremio uses the default profile.
Example: example-profile
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "fs.s3a.proxy.host","value": "proxyHost.example.com"}]
secure Body Boolean Optional
If set to enable a secure connection with SSL encryption between the S3 bucket and Dremio, the value is true (default). Otherwise, the value is false.
Example: true
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: fs.s3a.proxy.host
value Body String Optional
Value for the connection property.
Example: proxyHost.example.com
Oracle
Oracle Source config Object{
"config": {
"hostname": "172.25.0.10",
"port": "1521",
"instance": "xe",
"username": "exampleuser",
"password": "$DREMIO_EXISTING_VALUE$",
"authenticationType": "MASTER",
"fetchSize": 200,
"useSsl": false,
"useTimezoneAsRegion": true,
"includeSynonyms": false,
"useLdap": false,
"nativeEncryption": "ACCEPTED",
"useKerberos": false,
"maxIdleConns": 8,
"idleTimeSec": 60,
"mapDateToTimestamp": true,
"queryTimeoutSec": 0,
"propertyList": [
{
"name": "connect_timeout",
"value": "0"
}
]
}
}
Oracle config Parameters
hostname Body String
Oracle hostname.
Example: 172.25.0.10
port Body String
Oracle port number. Default is 1521.
Example: 1521
instance Body String
Name of the Oracle database.
Example: xe
username Body String
Username for authenticating with master credentials.
Example: exampleuser
password Body String
Password for authenticating with master credentials. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.
secretResourceUrl Body String
For secret-based authentication, the Amazon Resource Name (ARN) for the secret. To keep the secret secure, Dremio returns the secretResourceUrl value as $DREMIO_EXISTING_VALUE$ in API responses.
Example: arn:aws:secretsmanager:us-west-2:123456789012㊙️my-rds-secret-VNenFy
authenticationType Body String
Type of authentication to use to allow Dremio to connect to the Oracle source.
ANONYMOUS: No authentication is needed.MASTER: Use credentials from a master database user or use a secret resource URL.KERBEROS: Authenticate with Kerberos.
Example: MASTER
fetchSize Body Integer Optional
Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.
Example: 200
useSsl Body Boolean Optional
If the Oracle source is configured to use TLS for connections, set to true. Otherwise, set to false (default).
Example: false
useTimezoneAsRegion Body Boolean Optional
To use the timezone to set the connection region, set to true (default). Otherwise, set to false.
Example: true
includeSynonyms Body Boolean Optional
To include Oracle synonyms as datasets in the source, set to true. Otherwise, set to false (default).
Example: false
nativeEncryption Body String Optional
If using Oracle native encryption for the connection, the encryption setting to use. Default is ACCEPTED.
Enum: REJECTED, ACCEPTED, REQUESTED, REQUIRED
Example: ACCEPTED
useKerberos Body Boolean Optional
To use Kerberos for authentication, set to true. Otherwise, set to false (default).
Example: false
maxIdleConns Body Integer Optional
Maximum number of connections that can be idle at any given time. Default is 8.
Example: 8
idleTimeSec Body Integer Optional
Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.
Example: 60
mapDateToTimestamp Body Boolean Optional
If the DATE column should display values in timestamp format, set to true (default). Otherwise, set to false.
Example: true
queryTimeoutSec Body Integer Optional
Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.
Example: 0
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "connect_timeout","value": "0"}]
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: connect_timeout
value Body String Optional
Value for the connection property.
Example: 0
PostgreSQL
PostgreSQL Source config Object{
"config": {
"hostname": "172.25.0.10",
"port": "5432",
"databaseName": "postgres",
"username": "exampleuser",
"password": "$DREMIO_EXISTING_VALUE$",
"authenticationType": "MASTER",
"fetchSize": 200,
"useSsl": false,
"encryptionValidationMode": "CERTIFICATE_AND_HOSTNAME_VALIDATION",
"maxIdleConns": 8,
"idleTimeSec": 60,
"queryTimeoutSec": 0,
"propertyList": [
{
"name": "connect_timeout",
"value": "10"
}
]
}
}
PostgreSQLconfig Parameters
hostname Body String
PostgreSQL hostname.
Example: 172.25.0.10
port Body String
PostgreSQL port number. Default is 5432.
Example: 5432
databaseName Body String
Default PostgreSQL database for Dremio to use.
Example: postgres
username Body String
Username for authenticating with master credentials.
Example: exampleuser
password Body String
Password for authenticating with master credentials. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.
secretResourceUrl Body String
For secret-based authentication, the Amazon Resource Name (ARN) for the secret. To keep the secret secure, Dremio returns the secretResourceUrl value as $DREMIO_EXISTING_VALUE$ in API responses.
Example: arn:aws:secretsmanager:us-west-2:123456789012㊙️my-rds-secret-VNenFy
authenticationType Body String
Type of authentication to use to allow Dremio to connect to the PostgreSQL source.
ANONYMOUS: No authentication is needed.MASTER: Use credentials from a master database user or use a secret resource URL.
Example: MASTER
fetchSize Body Integer Optional
Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.
Example: 200
useSsl Body Boolean Optional
If the PostgreSQL source is configured to use TLS for connections, set to true. Otherwise, set to false.
Example: false
encryptionValidationMode Body String Optional
Method to use to validate data encryption for the source. The available settings map to PostgreSQL sslmode terminology as follows:
CERTIFICATE_AND_HOSTNAME_VALIDATION: verify-fullCERTIFICATE_ONLY_VALIDATION: verify-caNO_VALIDATION: require
Example: CERTIFICATE_AND_HOSTNAME_VALIDATION
maxIdleConns Body Integer Optional
Maximum number of connections that can be idle at any given time. Default is 8.
Example: 8
idleTimeSec Body Integer Optional
Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.
Example: 60
queryTimeoutSec Body Integer Optional
Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.
Example: 0
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "connect_timeout","value": "10&application_name=myapp"}]
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: connect_timeout
value Body String Optional
Value for the connection property.
Example: 10
Snowflake
Snowflake Source config Object{
"config": {
"schema": "myschema",
"hostname": "https://cxazbmq-pa44061.snowflakecomputing.com",
"database": "SNOWFLAKE_SAMPLE_DATA",
"role": "ACCOUNTADMIN",
"warehouse": "COMPUTE_WH",
"username": "exampleuser1",
"password": "$DREMIO_EXISTING_VALUE$",
"maxIdleConns": 8,
"idleTimeSec": 60,
"queryTimeoutSec": 0,
"propertyList": [
{
"name": "loginTimeout",
"value": "60"
}
]
}
}
Snowflake config Parameters
schema Body String Optional
Default Snowflake schema for Dremio to use for the specified database.
Example: myschema
hostname Body String
Account URL for the Snowflake source.
Example: https://cxazbmq-pa44061.snowflakecomputing.com
database Body String Optional
Default Snowflake database for Dremio to use.
Example: SNOWFLAKE_SAMPLE_DATA
role Body String Optional
Default access control role that Dremio should use in the Snowflake session initiated by the driver.
Example: ACCOUNTADMIN
warehouse Body String Optional
Snowflake virtual warehouse for Dremio to use.
Example: COMPUTE_WH
username Body String
Username for the Snowflake account.
Example: exampleuser1
password Body String
Password for the Snowflake account. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.
maxIdleConns Body Integer Optional
Maximum number of connections that can be idle at any given time. Default is 8.
Example: 8
idleTimeSec Body Integer Optional
Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.
Example: 60
queryTimeoutSec Body Integer Optional
Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.
Example: 0
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "loginTimeout","value": "60"}]
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: loginTimeout
value Body String Optional
Value for the connection property.
Example: 60
Teradata
Teradata Source config Object{
"config": {
"hostname": "teradata.drem.io",
"port": "1025",
"database": "dremio_source",
"username": "dremio",
"password": "$DREMIO_EXISTING_VALUE$",
"fetchSize": 200,
"showOnlyConnectionDatabase": false,
"useSsl": false,
"maxIdleConns": 8,
"idleTimeSec": 60,
"queryTimeoutSec": 0,
"propertyList": [
{
"name": "maxPoolSize",
"value": "10"
}
]
}
}
Teradata config Parameters
hostname Body String
Teradata hostname.
Example: teradata.drem.io
port Body String
Teradata port number. Default is 5432.
Example: 1025
database Body String Optional
Service name of the Teradata database.
Example: dremio_source
username Body String
Username for authentication.
Example: dremio
password Body String
Password for authentication. To keep the password secure, Dremio returns the password value as $DREMIO_EXISTING_VALUE$ in API responses.
fetchSize Body Integer Optional
Number of records to fetch at one time. Default is 200. If set to 0, Dremio automatically decides how many records to fetch.
Example: 200
showOnlyConnectionDatabase Body Boolean Optional
To hide all databases that the credentialed user can access except the specified connection database, set to true. Otherwise, set to false (default).
Example: false
useSsl Body Boolean Optional
If the Teradata source is configured to use TLS for connections, set to true. Otherwise, set to false (default).
Example: false
maxIdleConns Body Integer Optional
Maximum number of connections that can be idle at any given time. Default is 8.
Example: 8
idleTimeSec Body Integer Optional
Maximum time that a connection can remain idle before Dremio terminates it, in seconds. Default is 60.
Example: 60
queryTimeoutSec Body Integer Optional
Maximum time to allow for query execution, in seconds. When the query timeout expires, the connection returns to an idle state. Default is 0.
Example: 0
propertyList Body Array of Object Optional
Connection properties for the source. Each object in the propertyList includes the name of the property and the corresponding value to use for the property.
Example: [{"name": "maxPoolSize","value": "10"}]
Parameters of the propertyList Object
name Body String Optional
Name of the connection property.
Example: maxPoolSize
value Body String Optional
Value for the connection property.
Example: 10