> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gu1.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Ingestion Overview

> Ingest data into gu1 using custom schemas and field mappings to load entities, transactions, and events from CSV files, APIs, and event streams.

## What is Data Ingestion?

gu1's Data Ingestion system allows you to seamlessly import data from any source (CSV files, APIs, databases, or custom formats) by defining custom schemas and field mappings. This intelligent mapping system ensures your data is properly structured for risk analysis.

## How It Works

<Steps>
  <Step title="Define Your Schema">
    Create a custom schema that describes your data structure with field definitions, types, and validation rules.
  </Step>

  <Step title="Map Fields">
    Create field mappings that translate your data fields to gu1's unified entity model.
  </Step>

  <Step title="Transform Data">
    Apply transformations (formatting, calculations, conditionals) as data flows through the mapping.
  </Step>

  <Step title="Import Entities">
    Use the mapped schema to create entities via API or bulk upload.
  </Step>
</Steps>

## Key Features

<CardGroup cols={2}>
  <Card title="Custom Schemas" icon="table" href="/api-reference/data-ingestion/custom-schemas">
    Define your data structure with flexible field types and validation
  </Card>

  <Card title="Field Mappings" icon="arrows-left-right" href="/api-reference/data-ingestion/field-mappings">
    Map your fields to gu1's unified model with transformations
  </Card>

  <Card title="Smart Detection" icon="wand-magic-sparkles" href="/api-reference/data-ingestion/custom-schemas">
    Auto-detect field types, patterns, and suggested mappings
  </Card>

  <Card title="Bulk Processing" icon="layer-group" href="/en/api-reference/bulk-imports/overview">
    Import thousands of records efficiently with batch processing
  </Card>
</CardGroup>

## Schema Types

gu1 supports multiple schema types for different data sources:

| Type         | Description                   | Use Case                    |
| ------------ | ----------------------------- | --------------------------- |
| **database** | Relational database schemas   | Direct database integration |
| **api**      | API response structures       | Third-party API integration |
| **file**     | File formats (CSV, JSON, XML) | File-based imports          |
| **custom**   | Custom data structures        | Proprietary formats         |

## Schema Categories

Organize schemas by business domain:

<AccordionGroup>
  <Accordion icon="chart-line" title="Financial">
    Bank accounts, transactions, financial statements, payment data
  </Accordion>

  <Accordion icon="id-card" title="Identity">
    Personal information, identity documents, KYC data
  </Accordion>

  <Accordion icon="shield-check" title="Compliance">
    Sanctions lists, PEPs, adverse media, regulatory data
  </Accordion>

  <Accordion icon="money-bill-transfer" title="Transaction">
    Payment transactions, wire transfers, transaction history
  </Accordion>

  <Accordion icon="folder" title="General">
    Any other type of structured data
  </Accordion>
</AccordionGroup>

## Field Types

Supported field types for schema definition:

| Type        | Description      | Example                                                    |
| ----------- | ---------------- | ---------------------------------------------------------- |
| **string**  | Text data        | "Acme Corp", "[john@example.com](mailto:john@example.com)" |
| **number**  | Numeric values   | 1000, 99.99, -50                                           |
| **boolean** | True/false       | true, false                                                |
| **date**    | Date/timestamp   | "2025-10-03T12:00:00Z"                                     |
| **array**   | List of values   | \["tag1", "tag2"]                                          |
| **object**  | Nested structure | `{"city": "NYC", "country": "US"}`                         |

## Transformation Types

Apply transformations during field mapping:

<CardGroup cols={3}>
  <Card title="Direct" icon="arrow-right">
    Copy field as-is with no changes
  </Card>

  <Card title="Calculate" icon="calculator">
    Perform mathematical calculations
  </Card>

  <Card title="Format" icon="text">
    Format strings, dates, numbers
  </Card>

  <Card title="Conditional" icon="code-branch">
    Apply if/then logic based on conditions
  </Card>

  <Card title="Lookup" icon="magnifying-glass">
    Look up values from reference tables
  </Card>

  <Card title="Custom" icon="code">
    Custom JavaScript expressions
  </Card>
</CardGroup>

## Validation Rules

Ensure data quality with built-in validations:

```json theme={null}
{
  "constraints": {
    "minLength": 5,
    "maxLength": 100,
    "pattern": "^[A-Z0-9]+$",
    "enum": ["active", "inactive", "pending"]
  }
}
```

**Available Constraints:**

* `minLength` / `maxLength` - String length limits
* `min` / `max` - Numeric value ranges
* `pattern` - Regular expression validation
* `enum` - Allowed values list
* `required` - Field is mandatory

## Example: Banking Data Schema

Here's a complete example of defining a schema for banking customer data:

```json theme={null}
{
  "name": "Banking Customer Data",
  "version": "1.0.0",
  "type": "database",
  "category": "financial",
  "schemaData": {
    "fields": [
      {
        "name": "customer_id",
        "type": "string",
        "required": true,
        "description": "Unique customer identifier",
        "constraints": {
          "pattern": "^CUST[0-9]{8}$"
        }
      },
      {
        "name": "full_name",
        "type": "string",
        "required": true,
        "description": "Customer full legal name",
        "constraints": {
          "minLength": 2,
          "maxLength": 200
        }
      },
      {
        "name": "account_balance",
        "type": "number",
        "required": false,
        "description": "Current account balance in USD",
        "constraints": {
          "min": 0
        }
      },
      {
        "name": "risk_level",
        "type": "string",
        "required": true,
        "description": "Risk classification",
        "constraints": {
          "enum": ["low", "medium", "high", "critical"]
        }
      },
      {
        "name": "onboarding_date",
        "type": "date",
        "required": true,
        "description": "Date customer was onboarded"
      },
      {
        "name": "kyc_verified",
        "type": "boolean",
        "required": true,
        "description": "Whether KYC verification is complete"
      }
    ],
    "metadata": {
      "sourceFormat": "database",
      "encoding": "UTF-8"
    }
  }
}
```

## Best Practices

<AccordionGroup>
  <Accordion icon="lightbulb" title="Schema Design">
    * Use descriptive field names that match your source data
    * Include detailed descriptions for complex fields
    * Set appropriate validation constraints
    * Version your schemas (1.0.0, 1.1.0, etc.)
  </Accordion>

  <Accordion icon="arrows-spin" title="Field Mapping">
    * Start with direct mappings, add transformations as needed
    * Test mappings with sample data before bulk import
    * Document custom transformation logic
    * Handle null/missing values gracefully
  </Accordion>

  <Accordion icon="shield" title="Data Quality">
    * Validate data at the source before importing
    * Use strict mode for production environments
    * Monitor failed imports and validation errors
    * Implement data cleansing for known issues
  </Accordion>

  <Accordion icon="gauge-high" title="Performance">
    * Use bulk processing for large datasets (>1000 records)
    * Set appropriate batch sizes (100-1000 records)
    * Schedule imports during off-peak hours
    * Monitor processing times and adjust batch sizes
  </Accordion>
</AccordionGroup>

## Common Use Cases

<CardGroup cols={2}>
  <Card title="CSV File Import" icon="file-csv" href="/csv-import-guide">
    Import customer data from CSV files with automatic field detection
  </Card>

  <Card title="API Integration" icon="plug" href="/api-reference/data-ingestion/overview">
    Connect third-party APIs and sync data in real-time
  </Card>

  <Card title="Database Sync" icon="database" href="/DATABASE_SYNC_GUIDE">
    Synchronize data from your existing databases
  </Card>

  <Card title="Banking Onboarding" icon="building-columns" href="/use-cases/kyb/example">
    Complete KYB workflow with data mapping example
  </Card>
</CardGroup>

## API Endpoints

<CardGroup cols={2}>
  <Card title="Create Schema" icon="plus" href="/api-reference/data-ingestion/custom-schemas#create-schema">
    POST /custom-schemas
  </Card>

  <Card title="List Schemas" icon="list" href="/api-reference/data-ingestion/custom-schemas#list-schemas">
    GET /custom-schemas
  </Card>

  <Card title="Create Mapping" icon="plus" href="/api-reference/data-ingestion/field-mappings#create-mapping">
    POST /custom-schemas/mappings
  </Card>

  <Card title="Smart Detection" icon="wand-magic-sparkles" href="/api-reference/data-ingestion/custom-schemas">
    POST /custom-schemas/detect-fields
  </Card>
</CardGroup>

## Next Steps

<Steps>
  <Step title="Create Your First Schema">
    Follow the [Custom Schemas guide](/api-reference/data-ingestion/custom-schemas) to define your data structure
  </Step>

  <Step title="Map Your Fields">
    Learn how to map fields to gu1's model in the [Field Mappings guide](/api-reference/data-ingestion/field-mappings)
  </Step>

  <Step title="Import Data">
    Start importing entities using the [Entities API](/api-reference/entities/create)
  </Step>
</Steps>