> ## Documentation Index
> Fetch the complete documentation index at: https://unstructured-53-docs-243-plugins.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Snowflake
This page was recently updated. What do you think about it? [Let us know!](https://smart-forms.saasjet.com/external?token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnRLZXkiOiIzZGQ5OTEzYy04Mzg1LTNkNDAtYWQ1MS03NmM0Nzg5YjAxOGUiLCJpYXQiOjE3MTQ0MzI1ODU5MjF9.m5gHGCOHt_UgOs-JCdTdEHRcQUNftvLGWeKzK2o2pQ4).
Connect Snowflake to your preprocessing pipeline, and use the Unstructured Ingest CLI or the Unstructured Ingest Python library to batch process all your documents and store structured outputs locally on your filesystem.
The requirements are as follows.
* A Snowflake [account](https://signup.snowflake.com/) and its account identifier.
To get the identifier for the current Snowflake account:
1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
2. In Snowsight, on the navigation menu, click your username, and then click **Account > View account details**.
3. On the **Account** tab, note the value of the **Account Identifier** field.
Alternatively, the following Snowflake query returns the current account's identifier:
```text theme={null}
SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME() AS "Account Identifier"
```
* The Snowflake [user's login name (not its username) and its password](https://docs.snowflake.com/user-guide/admin-user-management#creating-users) in the account.
To view the login name for a user:
1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
2. In Snowsight, on the navigation menu, click **Admin > Users & Roles**.
3. On the **Users** tab, in the list of available users, click the name of the target user.
4. In the **About** tile, note the **Login Name** for the user.
Alternatively, the following Snowflake query returns information about the user with the username of ``, including their `login_name` value representing their login name:
```text theme={null}
SHOW USERS LIKE '';
```
* The name of the Snowflake [role](https://docs.snowflake.com/sql-reference/sql/create-role) that the user belongs to and that also has sufficient access to the Snowflake database, schema, table, and host.
* To create a database in Snowflake, the role needs to be granted `CREATE DATABASE` privilege at the current account level; and `USAGE` privilege on the warehouse that is used to create the database.
* To create a schema in a database in Snowflake, the role needs to be granted `USAGE` privilege on the database and the warehouse that is used to create the schema; and `CREATE SCHEMA` on the database.
* To create a table in a schema in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to create the table; and `CREATE TABLE` on the schema.
* To write to a table in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to write to the table; and `INSERT` on the table.
* To read from a table in Snowflake, the role needs to be granted `USAGE` privilege on the database and schema and the warehouse that is used to write to the table; and `SELECT` on the table.
To view a list of available roles in the current Snowflake account:
1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
2. In Snowsight, on the navigation menu, click **Admin > Users & Roles**.
3. Click the **Roles** tab.
Alternatively, the following Snowflake query returns a list of available roles in the current account:
```text theme={null}
SHOW ROLES;
```
[Grant privileges to a role](https://docs.snowflake.com/sql-reference/sql/grant-privilege). [Learn more](https://docs.snowflake.com/user-guide/security-access-control-privileges).
* The Snowflake warehouse's [hostname and its port number](https://docs.snowflake.com/sql-reference/functions/system_allowlist) in the account.
To view a list of available warehouses in the current Snowflake account:
1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
2. In Snowsight, on the navigation menu, click **Admin > Warehouses**. This view does not provide access to the warehouses' hostnames or port numbers. To get this information, you must run a Snowflake query.
The following Snowflake query returns a list of available warehouse types, hostnames, and port numbers in the current account. Look for the row with a `type` of `SNOWFLAKE_DEPLOYMENT`:
```text theme={null}
SELECT t.VALUE:type::VARCHAR as type,
t.VALUE:host::VARCHAR as host,
t.VALUE:port as port
FROM TABLE(FLATTEN(input => PARSE_JSON(SYSTEM$ALLOWLIST()))) AS t;
```
* The name of the Snowflake [database](https://docs.snowflake.com/sql-reference/sql/create-database) in the account.
To view a list of available databases in the current Snowflake account:
1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
2. In Snowsight, on the navigation menu, click **Data > Databases**.
Alternatively, the following Snowflake query returns a list of available databases in the current account:
```text theme={null}
SHOW DATABASES;
```
* The name of the [schema](https://docs.snowflake.com/sql-reference/sql/create-schema) in the database.
To view a list of available schemas for a database in the current Snowflake account:
1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
2. In Snowsight, on the navigation menu, click **Data > Databases**.
3. Expand the name of the target database.
Alternatively, the following Snowflake query returns a list of available schemas in the current account:
```text theme={null}
SHOW SCHEMAS;
```
The following Snowflake query returns a list of available schemas for the database named `` in the current account:
```text theme={null}
SHOW SCHEMAS IN DATABASE ;
```
* The name of the [table](https://docs.snowflake.com/sql-reference/sql/create-table) in the schema.
To view a list of available tables for a schema in a database in the current Snowflake account:
1. Log in to [Snowsight](https://docs.snowflake.com/user-guide/ui-snowsight-homepage) with your Snowflake account.
2. In Snowsight, on the navigation menu, click **Data > Databases**.
3. Expand the name of the database that contains the target schema.
4. Expand the name of the target schema.
5. Expand **Tables**.
Alternatively, the following Snowflake query returns a list of available tables for the schema named `` in the datbase named
`` in the current account:
```text theme={null}
SHOW TABLES IN SCHEMA .;
```
Snowflake requires the target table to have a defined schema before Unstructured can write to the table. The recommended table
schema for Unstructured is as follows:
```sql SQL theme={null}
CREATE TABLE ELEMENTS (
ID VARCHAR(36) NOT NULL DEFAULT UUID_STRING(),
RECORD_ID VARCHAR,
ELEMENT_ID VARCHAR,
TEXT VARCHAR,
EMBEDDINGS ARRAY,
TYPE VARCHAR,
SYSTEM VARCHAR,
LAYOUT_WIDTH NUMBER,
LAYOUT_HEIGHT NUMBER,
POINTS VARCHAR,
URL VARCHAR,
VERSION VARCHAR,
DATE_CREATED TIMESTAMP_TZ,
DATE_MODIFIED TIMESTAMP_TZ,
DATE_PROCESSED TIMESTAMP_TZ,
PERMISSIONS_DATA VARCHAR,
RECORD_LOCATOR VARCHAR,
CATEGORY_DEPTH NUMBER,
PARENT_ID VARCHAR,
ATTACHED_FILENAME VARCHAR,
FILETYPE VARCHAR,
LAST_MODIFIED TIMESTAMP_TZ,
FILE_DIRECTORY VARCHAR,
FILENAME VARCHAR,
LANGUAGES ARRAY,
PAGE_NUMBER VARCHAR,
LINKS VARCHAR,
PAGE_NAME VARCHAR,
LINK_URLS ARRAY,
LINK_TEXTS ARRAY,
SENT_FROM ARRAY,
SENT_TO ARRAY,
SUBJECT VARCHAR,
SECTION VARCHAR,
HEADER_FOOTER_TYPE VARCHAR,
EMPHASIZED_TEXT_CONTENTS ARRAY,
EMPHASIZED_TEXT_TAGS ARRAY,
TEXT_AS_HTML VARCHAR,
REGEX_METADATA VARCHAR,
DETECTION_CLASS_PROB NUMBER,
PRIMARY KEY (ID),
PARTITIONER_TYPE VARCHAR
);
```
* The name of the column in the table that uniquely identifies each record (for example, `RECORD_ID`).
The Snowflake connector dependencies:
```bash CLI, Python theme={null}
pip install "unstructured-ingest[snowflake]"
```
You might also need to install additional dependencies, depending on your needs. [Learn more](/ingestion/ingest-dependencies).
These environment variables:
* `SNOWFLAKE_ACCOUNT` - The ID of the target Snowflake account, represented by `--account` (CLI) or `account` (Python).
* `SNOWFLAKE_USER` - The name of the target Snowflake user, represented by `--user` (CLI) or `user` (Python).
* `SNOWFLAKE_PASSWORD` - The user's password, represented by `--password` (CLI) or `password` (Python).
* `SNOWFLAKE_ROLE` - The target role for the user, represented by `--role` (CLI) or `role` (Python).
* `SNOWFLAKE_HOST` - The hostname for the target Snowflake warehouse, represented by `--host` (CLI) or `host` (Python).
* `SNOWFLAKE_PORT` - The warehouse's port number, represented by `--port` (CLI) or `port` (Python). The default is `443` if not otherwise specified.
* `SNOWFLAKE_DATABASE` - The name of the target Snowflake database, represented by `--database` (CLI) or `database` (Python).
* `SNOWFLAKE_SCHEMA` - The name of the target schema in the database, represented by `--schema` (CLI) or `schema` (Python).
* `SNOWFLAKE_TABLE` - The name of the target table in the schema, represented by `--table-name` (CLI) or `table_name` (Python). For the destination connector, the default is `elements` if not otherwise specified.
* `SNOWFLAKE_RECORD_ID_KEY` - The name of the column in the table that uniquely identifies each record, represented by:
* For the source connector, `--id-column` (CLI) or `id_column` (Python).
* For the destination connector, `--record-id-key` (CLI) or `record_id_key` (Python). For the destination connector, the default is `record_id` if not otherwise specified.
Now call the Unstructured Ingest CLI or the Unstructured Ingest Python library. The destination connector can be any of the ones supported. This example uses the local destination connector:
This example sends data to Unstructured for processing by default. To process data locally instead, see the instructions at the end of this page.
```bash CLI theme={null}
#!/usr/bin/env bash
unstructured \
snowflake \
--account $SNOWFLAKE_ACCOUNT \
--user $SNOWFLAKE_USER \
--password $SNOWFLAKE_PASSWORD \
--role $SNOWFLAKE_ROLE \
--host $SNOWFLAKE_HOST \
--port $SNOWFLAKE_PORT \
--database $SNOWFLAKE_DATABASE \
--schema $SNOWFLAKE_SCHEMA \
--batch-size 50 \
--table-name $SNOWFLAKE_TABLE \
--id-column $SNOWFLAKE_RECORD_ID_KEY \
--download-dir $LOCAL_FILE_DOWNLOAD_DIR\
--partition-by-api \
--api-key $UNSTRUCTURED_API_KEY \
--partition-endpoint $UNSTRUCTURED_API_URL \
--output-dir $LOCAL_FILE_OUTPUT_DIR
```
```python Python Ingest theme={null}
import os
from unstructured_ingest.pipeline.pipeline import Pipeline
from unstructured_ingest.interfaces import ProcessorConfig
from unstructured_ingest.processes.connectors.sql.snowflake import (
SnowflakeIndexerConfig,
SnowflakeConnectionConfig,
SnowflakeAccessConfig,
SnowflakeDownloaderConfig
)
from unstructured_ingest.processes.connectors.local import LocalUploaderConfig
from unstructured_ingest.processes.partitioner import PartitionerConfig
from unstructured_ingest.processes.chunker import ChunkerConfig
from unstructured_ingest.processes.embedder import EmbedderConfig
# Chunking and embedding are optional.
if __name__ == "__main__":
Pipeline.from_configs(
context=ProcessorConfig(),
indexer_config=SnowflakeIndexerConfig(
table_name=os.getenv("SNOWFLAKE_TABLE"),
id_column=os.getenv("SNOWFLAKE_RECORD_ID_KEY"),
batch_size=100
),
downloader_config=SnowflakeDownloaderConfig(download_dir=os.getenv("LOCAL_FILE_DOWNLOAD_DIR")),
source_connection_config=SnowflakeConnectionConfig(
access_config=SnowflakeAccessConfig(
password=os.getenv("SNOWFLAKE_PASSWORD")
),
account=os.getenv("SNOWFLAKE_ACCOUNT"),
user=os.getenv("SNOWFLAKE_USER"),
host=os.getenv("SNOWFLAKE_HOST"),
port=os.getenv("SNOWFLAKE_PORT"),
database=os.getenv("SNOWFLAKE_DATABASE"),
schema=os.getenv("SNOWFLAKE_SCHEMA"),
role=os.getenv("SNOWFLAKE_ROLE")
),
partitioner_config=PartitionerConfig(
partition_by_api=True,
api_key=os.getenv("UNSTRUCTURED_API_KEY"),
partition_endpoint=os.getenv("UNSTRUCTURED_API_URL"),
additional_partition_args={
"split_pdf_page": True,
"split_pdf_allow_failed": True,
"split_pdf_concurrency_level": 15
}
),
chunker_config=ChunkerConfig(chunking_strategy="by_title"),
embedder_config=EmbedderConfig(embedding_provider="huggingface"),
uploader_config=LocalUploaderConfig(output_dir=os.getenv("LOCAL_FILE_OUTPUT_DIR"))
).run()
```
For the Unstructured Ingest CLI and the Unstructured Ingest Python library, you can use the `--partition-by-api` option (CLI) or `partition_by_api` (Python) parameter to specify where files are processed:
* To do local file processing, omit `--partition-by-api` (CLI) or `partition_by_api` (Python), or explicitly specify `partition_by_api=False` (Python).
Local file processing does not use an Unstructured API key or API URL, so you can also omit the following, if they appear:
* `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
* `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
* The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`
* To send files to the [Unstructured Partition Endpoint](/api-reference/partition/overview) for processing, specify `--partition-by-api` (CLI) or `partition_by_api=True` (Python).
Unstructured also requires an Unstructured API key and API URL, by adding the following:
* `--api-key $UNSTRUCTURED_API_KEY` (CLI) or `api_key=os.getenv("UNSTRUCTURED_API_KEY")` (Python)
* `--partition-endpoint $UNSTRUCTURED_API_URL` (CLI) or `partition_endpoint=os.getenv("UNSTRUCTURED_API_URL")` (Python)
* The environment variables `UNSTRUCTURED_API_KEY` and `UNSTRUCTURED_API_URL`, representing your API key and API URL, respectively.
You must specify the API URL only if you are not using the default API URL for Unstructured Ingest, for example, if you are using a version of the Unstructured API that is hosted on your own compute infrastructure.
The default API URL for Unstructured Ingest is `https://api.unstructuredapp.io/general/v0/general`, which is the API URL for the [Unstructured Partition Endpoint](/api-reference/partition/overview).
If you do not have an API key, [get one now](/api-reference/partition/overview).
If the Unstructured API is hosted on your own compute infrastructure, the process
for generating Unstructured API keys, and the Unstructured API URL that you use, are different.
For details, contact Unstructured Sales at
[sales@unstructured.io](mailto:sales@unstructured.io).