Skip to content

Granit.DataExchange

Granit.DataExchange provides a complete import/export pipeline for tabular data. Imports follow a guided flow: upload a file, preview headers, receive intelligent column mapping suggestions, confirm, then execute with batched persistence and detailed error reporting. Exports use a whitelist-based field definition, optional presets, and automatic background dispatch for large datasets. Both pipelines integrate with Wolverine for durable outbox-backed execution when installed.

  • DirectoryGranit.DataExchange/ Core: import/export pipelines, fluent definitions, mapping suggestion engine
    • Granit.DataExchange.Csv CSV parser (Sep SIMD) and writer (semicolon separator)
    • Granit.DataExchange.Excel Excel parser (Sylvan.Data.Excel) and writer (ClosedXML)
    • Granit.DataExchange.EntityFrameworkCore DataExchangeDbContext, EF executor, identity resolvers, stores
    • Granit.DataExchange.Endpoints REST endpoints for import/export operations
    • Granit.DataExchange.Wolverine Replaces Channel dispatchers with Wolverine outbox-backed dispatchers
PackageRoleDepends on
Granit.DataExchangeImport/export pipelines, mapping suggestions, fluent definitionsGranit.Timing, Granit.Validation
Granit.DataExchange.CsvSep-based CSV parser, semicolon CSV writerGranit.DataExchange
Granit.DataExchange.ExcelSylvan streaming Excel reader, ClosedXML writerGranit.DataExchange
Granit.DataExchange.EntityFrameworkCoreDataExchangeDbContext, EF executor, identity resolversGranit.DataExchange, Granit.Persistence
Granit.DataExchange.Endpoints19 REST endpoints (import + export + metadata)Granit.DataExchange, Granit.Authorization
Granit.DataExchange.WolverineOutbox-backed import/export dispatchGranit.DataExchange, Granit.Wolverine
graph TD
    DX[Granit.DataExchange] --> T[Granit.Timing]
    DX --> V[Granit.Validation]
    CSV[Granit.DataExchange.Csv] --> DX
    XLS[Granit.DataExchange.Excel] --> DX
    EF[Granit.DataExchange.EntityFrameworkCore] --> DX
    EF --> P[Granit.Persistence]
    EP[Granit.DataExchange.Endpoints] --> DX
    EP --> A[Granit.Authorization]
    WV[Granit.DataExchange.Wolverine] --> DX
    WV --> W[Granit.Wolverine]
[DependsOn(
typeof(GranitDataExchangeEntityFrameworkCoreModule),
typeof(GranitDataExchangeEndpointsModule),
typeof(GranitDataExchangeCsvModule),
typeof(GranitDataExchangeExcelModule),
typeof(GranitDataExchangeWolverineModule))]
public class AppModule : GranitModule
{
public override void ConfigureServices(ServiceConfigurationContext context)
{
// Register import definitions
context.Services.AddImportDefinition<Patient, PatientImportDefinition>();
// Register export definitions
context.Services.AddExportDefinition<Patient, PatientExportDefinition>();
}
}
// Map endpoints in Program.cs
app.MapDataExchangeEndpoints();
// Or with custom prefix and role
app.MapDataExchangeEndpoints(opts =>
{
opts.RoutePrefix = "admin/data-exchange";
opts.RequiredRole = "ops-team";
});
{
"DataExchange": {
"DefaultMaxFileSizeMb": 50,
"DefaultBatchSize": 500,
"FuzzyMatchThreshold": 0.8
}
}
PropertyDefaultDescription
DefaultMaxFileSizeMb50Max upload size (overridable per definition)
DefaultBatchSize500Rows per SaveChanges batch
FuzzyMatchThreshold0.8Minimum Levenshtein similarity for fuzzy tier (0.0 - 1.0)
flowchart LR
    A[Upload file] --> B[Extract headers]
    B --> C[Preview rows]
    C --> D["Suggest mappings<br/>4-tier"]
    D --> E["User confirms<br/>mappings"]
    E --> F[Parse rows]
    F --> G[Map to entities]
    G --> H[Validate rows]
    H --> I[Resolve identity]
    I --> J["Execute batch<br/>INSERT / UPDATE"]
    J --> K[Report + correction file]

Each entity requires an ImportDefinition<T> that declares importable properties using a fluent API. Only explicitly declared properties are available for column mapping (whitelist pattern).

public sealed class PatientImportDefinition : ImportDefinition<Patient>
{
public override string Name => "Acme.PatientImport";
protected override void Configure(ImportDefinitionBuilder<Patient> builder)
{
builder
.HasBusinessKey(p => p.Niss)
.Property(p => p.Niss, p => p.DisplayName("NISS").Required())
.Property(p => p.LastName, p => p.DisplayName("Last name").Required())
.Property(p => p.FirstName, p => p.DisplayName("First name").Required())
.Property(p => p.Email, p => p
.DisplayName("Email")
.Aliases("Courriel", "E-mail", "Mail"))
.Property(p => p.BirthDate, p => p
.DisplayName("Date of birth")
.Format("dd/MM/yyyy"))
.ExcludeOnUpdate(p => p.Niss);
}
}

Property configuration options:

MethodDescription
.DisplayName(string)User-facing label (used in preview UI and mapping suggestions)
.Description(string)Sent to the AI mapping service as field metadata
.Aliases(params string[])Alternative names for exact and fuzzy matching
.Required(bool)Import-level required validation (independent of entity [Required])
.Format(string)Expected format for type conversion (e.g. "dd/MM/yyyy")

Identity resolution:

MethodDescription
.HasBusinessKey(p => p.Niss)Single natural key for INSERT vs UPDATE resolution
.HasCompositeKey(p => p.Code, p => p.Year)Multi-column business key
.HasExternalId()External ID column for cross-system identity mapping

Parent/child import:

builder
.GroupBy("InvoiceNumber")
.Property(p => p.InvoiceNumber, p => p.Required())
.Property(p => p.CustomerName)
.HasMany(p => p.Lines, child =>
{
child.Property(l => l.ProductCode, p => p.Required());
child.Property(l => l.Quantity);
child.Property(l => l.UnitPrice);
});

When headers are extracted from the uploaded file, the mapping suggestion service runs four tiers in order. Columns matched by a higher-confidence tier are excluded from lower tiers:

flowchart TD
    H[Source column headers] --> T1
    T1["Tier 1: Saved mappings<br/>Previously confirmed by user"] --> T2
    T2["Tier 2: Exact match<br/>Property name, display name, aliases"] --> T3
    T3["Tier 3: Fuzzy match<br/>Levenshtein distance >= threshold"] --> T4
    T4["Tier 4: Semantic / AI<br/>Header metadata only, GDPR-safe"] --> R[Suggested mappings]
TierConfidenceSource
SavedMappingConfidence.SavedPreviously confirmed mappings stored in database
ExactMappingConfidence.ExactCase-insensitive match on property name, display name, or aliases
FuzzyMappingConfidence.FuzzyLevenshtein similarity above FuzzyMatchThreshold
SemanticMappingConfidence.SemanticAI-backed service (opt-in, only header metadata sent)
StatusDescription
CreatedFile uploaded, job created
PreviewedHeaders extracted, preview and mapping suggestions generated
MappedColumn mappings confirmed by the user
ExecutingImport running (background handler)
CompletedAll rows imported successfully
PartiallyCompletedSome rows failed, others succeeded
FailedImport failed entirely
CancelledCancelled by the user
new ImportExecutionOptions
{
BatchSize = 500, // Rows per SaveChanges batch
DryRun = true, // Full pipeline with transaction rollback
ErrorBehavior = ImportErrorBehavior.SkipErrors,
}
Error behaviorDescription
FailFastStop immediately on the first error
SkipErrorsSkip errored rows, continue processing (default)
CollectAllProcess all rows, collect all errors without stopping

After an import with SkipErrors or CollectAll, a downloadable CSV correction file is generated. It contains only the failed rows with an additional error message column. Users can fix the rows and re-upload the corrected file.

flowchart LR
    A[Request export] --> B{Row count?}
    B -- "threshold or less" --> C[Synchronous export]
    B -- ">threshold" --> D[Background job]
    C --> E[Query data source]
    D --> E
    E --> F[Project fields]
    F --> G[Write CSV / Excel]
    G --> H[Store blob]
    H --> I[Download link]

Each entity requires an ExportDefinition<T> with a field whitelist. Only declared fields can appear in the output:

public sealed class PatientExportDefinition : ExportDefinition<Patient>
{
public override string Name => "Acme.PatientExport";
public override string? QueryDefinitionName => "Acme.Patients";
protected override void Configure(ExportDefinitionBuilder<Patient> builder)
{
builder
.IncludeBusinessKey()
.Field(p => p.LastName, f => f.Header("Last name"))
.Field(p => p.FirstName, f => f.Header("First name"))
.Field(p => p.Email)
.Field(p => p.BirthDate, f => f
.Header("Date of birth")
.Format("dd/MM/yyyy"))
.Field(p => p.Company, c => c.Name, f => f.Header("Company"));
}
}

Field configuration options:

MethodDescription
.Header(string)Column header name in the exported file
.Format(string)Display format (e.g. "dd/MM/yyyy", "#,##0.00")
.Order(int)Column order (lower values first)

Definition-level options:

MethodDescription
.IncludeId()Include entity Id column for roundtrip import compatibility
.IncludeBusinessKey()Include business key columns from the matching import definition

Navigation fields use a two-argument Field() overload for dot-notation traversal. The developer must ensure the corresponding Include() is present in the IExportDataSource<T> implementation.

Presets are named field selections that users can save and reuse. They are stored in the database via IExportPresetReader / IExportPresetWriter. The REST API exposes CRUD operations under /metadata/presets/.

StatusDescription
QueuedJob created and queued for background execution
ExportingExport currently being generated
CompletedFile available for download
FailedExport failed

When QueryDefinitionName is set on an export definition, the export pipeline delegates filtering and sorting to IQueryEngine<T> from Granit.Querying. This reuses the same whitelist-based filtering pipeline as the grid view — the user’s active filters are applied to the export.

FormatParser (import)Writer (export)Package
CSVSep (SIMD-accelerated)Semicolon separator (EU locale)Granit.DataExchange.Csv
Excel (.xlsx)Sylvan.Data.Excel (streaming)ClosedXMLGranit.DataExchange.Excel

All endpoints require authorization. Import endpoints use the DataExchange.Imports.Execute permission, export endpoints use DataExchange.Exports.Execute.

MethodPathDescription
GET/jobsList import jobs
POST/Upload file (creates import job)
POST/{jobId}/previewExtract headers and generate mapping suggestions
PUT/{jobId}/mappingsConfirm column mappings
POST/{jobId}/executeExecute the import
POST/{jobId}/dry-runFull pipeline with transaction rollback
GET/{jobId}Get import job status
DELETE/{jobId}Cancel import job
GET/{jobId}/reportGet import report (success/error counts, row details)
GET/{jobId}/correction-fileDownload CSV with failed rows and error messages
MethodPathDescription
GET/export/jobsList export jobs
POST/export/jobsCreate and execute export
GET/export/jobs/{id}Get export job status
GET/export/jobs/{id}/downloadDownload exported file
MethodPathDescription
GET/metadata/definitionsList registered export definitions
GET/metadata/definitions/{name}/fieldsList available fields for a definition
GET/metadata/presets/{definitionName}List saved presets for a definition
POST/metadata/presetsSave a field selection preset
DELETE/metadata/presets/{definitionName}/{presetName}Delete a preset

Granit.DataExchange.EntityFrameworkCore provides:

  • DataExchangeDbContext with entities for import jobs, export jobs, saved mappings, external ID mappings, and export presets.
  • EfImportExecutor — batched INSERT/UPDATE executor with SaveChanges per batch.
  • Identity resolvers: BusinessKeyResolver, CompositeKeyResolver — query the database to determine whether each row is an INSERT or UPDATE.
EntityPurpose
ImportJobEntityTracks import job lifecycle and metadata
ExportJobEntityTracks export job lifecycle and file location
SavedMappingEntityPersists confirmed column mappings for reuse (Tier 1)
ExternalIdMappingEntityMaps external identifiers to internal entity IDs
ExportPresetEntityNamed field selection presets

Without Wolverine, import and export commands dispatch via in-memory Channel<T> — messages are lost on crash. Adding Granit.DataExchange.Wolverine replaces both dispatchers with Wolverine’s IMessageBus for durable outbox-backed execution:

ServiceWithout WolverineWith Wolverine
IImportCommandDispatcherChannelImportCommandDispatcherWolverineImportCommandDispatcher
IExportCommandDispatcherChannelExportCommandDispatcherWolverineExportCommandDispatcher
IDataExchangeEventPublisherNo-opWolverineDataExchangeEventPublisher
CategoryKey typesPackage
ModuleGranitDataExchangeModule, GranitDataExchangeCsvModule, GranitDataExchangeExcelModule, GranitDataExchangeEntityFrameworkCoreModule, GranitDataExchangeEndpointsModule, GranitDataExchangeWolverineModule---
Import pipelineIImportOrchestrator, IMappingSuggestionService, IFileParser, IDataMapper, IRowValidator, IImportExecutorGranit.DataExchange
Import definitionImportDefinition<T>, ImportDefinitionBuilder<T>, PropertyMappingBuilderGranit.DataExchange
Import identityIRecordIdentityResolver, RecordIdentity, RecordOperationGranit.DataExchange
Import reportingImportReport, ImportProgress, ICorrectionFileGeneratorGranit.DataExchange
Export pipelineIExportOrchestrator, IExportWriter, IExportDataSource<T>Granit.DataExchange
Export definitionExportDefinition<T>, ExportDefinitionBuilder<T>, ExportFieldBuilderGranit.DataExchange
Export presetsIExportPresetReader, IExportPresetWriterGranit.DataExchange
MappingMappingConfidence, ImportColumnMapping, ISemanticMappingServiceGranit.DataExchange
OptionsImportOptions, ExportOptions, ImportExecutionOptionsGranit.DataExchange
PermissionsDataExchangePermissions.Imports.Execute, DataExchangePermissions.Exports.ExecuteGranit.DataExchange.Endpoints
ExtensionsAddImportDefinition<T, TDef>(), AddExportDefinition<T, TDef>(), AddSemanticMappingService<T>(), MapDataExchangeEndpoints()---