GitLab

GitLab ETL connector for data replication

Snapshot

Features	Details
Release Status	Released
Source API Version	V4
Table Selection	Yes
Column Selection	Yes
Edit Integration	Yes
Replication Type Selection	No
Authentication Parameters	GitLab Company Name GitLab API Key
Replication Type	Full Replication Key Based Incremental
Replication Key	Date
Suggested Replication Frequency	1 hour

Tables/APIs supported

Projects	Groups
Epics	Project Milestones
Group Milestones	Releases
Tags	Todos
Issues	Comments
Commits	Merge Requests
Branches	Users

Integrate GitLab with Daton

1. Sign in to Daton

2. Select GitLab from the list of Integrations

3. Provide Integration Name, Replication Frequency, and History. Integration name would be used in creating tables for the integration and cannot be changed later

4. Provide your GitLab company name and API Key

5. Post successful authentication, you will be prompted to choose from the list of available tables

6. Then select all required fields for each table

7. Submit the integration

Workflow

1. Integrations would be in Pending state initially and will be moved to Active state as soon as the first job loads data successfully on to the configured warehouse

2. Users would be able to edit/pause/re-activate/delete integration anytime

3. Users can view job status and process logs from the integration details page by clicking on the integration name from the active list

GitLab Data

Projects

Purpose: This API is used to fetch all the projects in your GitLab Account

Source API Documentation: https://docs.gitlab.com/ee/api/projects.html

Replication: Key Based Incremental

Replication Key: last_activity_after, last_activity_before

Fields

Name	Data Type
id	NUMERIC
description	STRING
name	STRING
name_with_namespace	STRING
path	STRING
path_with_namespace	STRING
created_at	TIMESTAMP
default_branch	STRING
tag_list	STRING
topics	STRING
ssh_url_to_repo	STRING
http_url_to_repo	STRING
web_url	STRING
readme_url	STRING
avatar_url	STRING
forks_count	NUMERIC
star_count	NUMERIC
last_activity_at	TIMESTAMP
namespace	RECORD

Groups

Purpose: This API is used to fetch all the groups

Source API Documentation: https://docs.gitlab.com/ee/api/groups.html

Replication: Full Replication

Replication Key: NA

Fields

Name	Data Type
id	NUMERIC
name	STRING
path	STRING
description	STRING
visibility	STRING
share_with_group_lock	BOOLEAN
require_two_factor_authentication	BOOLEAN
two_factor_grace_period	NUMERIC
project_creation_level	STRING
subgroup_creation_level	STRING
lfs_enabled	BOOLEAN
default_branch_protection	NUMERIC
avatar_url	STRING
web_url	STRING
request_access_enabled	BOOLEAN
full_name	STRING
full_path	STRING
file_template_project_id	NUMERIC
created_at	TIMESTAMP

Epics (GitLab Premium Users Only)

Purpose: This API is used to fetch all the Epics under the groups. This is available only to GitLab Premium Self-managed, GitLab Premium SaaS and higher tiers

Source API Documentation: https://docs.gitlab.com/ee/api/epics.html

Replication: Key Based Incremental

Replication Key: updated_after, updated_before

Fields

Name	Data Type
id	NUMERIC
iid	NUMERIC
group_id	NUMERIC
parent_id	NUMERIC
parent_iid	NUMERIC
title	STRING
description	STRING
state	STRING
confidential	STRING
web_url	STRING
reference	STRING
references	RECORD
author	RECORD
start_date_is_fixed	BOOLEAN
end_date	DATE
due_date	DATE
due_date_is_fixed	BOOLEAN
due_date_from_milestones	DATE
due_date_from_inherited_source	DATE
created_at	TIMESTAMP
updated_at	TIMESTAMP
closed_at	TIMESTAMP
upvotes	NUMERIC
downvotes	NUMERIC
color	STRING
_links	RECORD

Project Milestones

Purpose: This API is used to fetch the milestones of all the projects

Source API Documentation: https://docs.gitlab.com/ee/api/milestones.html

Replication: Key Based Incremental

Replication Key: last_activity_after, last_activity_before

Fields

Name	Data Type
id	NUMERIC
iid	NUMERIC
project_id	NUMERIC
title	STRING
description	STRING
due_date	DATE
start_date	DATE
state	STRING
updated_at	TIMESTAMP
created_at	TIMESTAMP
expired	BOOLEAN

Group Milestones

Purpose: This API is used to fetch milestones of all the groups

Source API Documentation: https://docs.gitlab.com/ee/api/group_milestones.html

Replication: Full Replication

Replication Key: NA

Fields

Name	Data Type
id	NUMERIC
iid	NUMERIC
group_id	NUMERIC
title	STRING
description	STRING
due_date	DATE
start_date	DATE
state	STRING
updated_at	TIMESTAMP
created_at	TIMESTAMP
expired	BOOLEAN
web_url	STRING

Releases

Purpose: This API is used to fetch the releases project wise

Source API Documentation: https://docs.gitlab.com/ee/api/releases/

Replication: Key Based Incremental

Replication Key: last_activity_after, last_activity_before

Fields

Name	Data Type
tag_name	STRING
description	STRING
name	STRING
created_at	TIMESTAMP
released_at	TIMESTAMP
author	RECORD
commit	RECORD
milestones	RECORD
commit_path	STRING
tag_path	STRING
assets	RECORD
evidences	RECORD

Tags

Purpose: This API is used to get a list of repository tags from a project

Source API Documentation: https://docs.gitlab.com/ee/api/tags.html

Replication: Key Based Incremental

Replication Key: last_activity_after, last_activity_before

Fields

Name	Data Type
commit	RECORD
release	RECORD
name	STRING
target	STRING
protected	BOOLEAN
project_id	STRING

Todos

Purpose: This API is used to fetch all to-dos that are done and pending

Source API Documentation: https://docs.gitlab.com/ee/api/todos.html

Replication: Full Replication

Replication Key: NA

Fields

Name	Data Type
id	NUMERIC
project	RECORD
author	RECORD
action_name	STRING
target_type	STRING
target	RECORD
target_url	STRING
body	STRING
state	STRING
created_at	TIMESTAMP
updated_at	TIMESTAMP

Issues

Purpose: This API is used to fetch all the issues in the account

Source API Documentation: https://docs.gitlab.com/ee/api/issues.html

Replication: Key Based Incremental

Replication Key: updated_after, updated_before

Fields

Name	Data Type
state	STRING
description	STRING
author	RECORD
milestone	RECORD
project_id	NUMERIC
assignees	RECORD
assignee	RECORD
type	STRING
updated_at	TIMESTAMP
id	NUMERIC
title	STRING
created_at	TIMESTAMP
iid	NUMERIC
labels	STRING
upvotes	NUMERIC
downvotes	NUMERIC
merge_requests_count	NUMERIC
user_notes_count	NUMERIC
due_date	DATE
web_url	STRING
references	RECORD
time_stats	RECORD
has_tasks	BOOLEAN
task_status	STRING
confidential	BOOLEAN
discussion_locked	BOOLEAN
issue_type	STRING
severity	STRING
_links	RECORD
task_completion_status	RECORD

Comments

Purpose: This API is used to fetch all comments in issues, epics, snippets and merge requests

Source API Documentation: https://docs.gitlab.com/ee/api/notes.html

Replication: Key Based Incremental

Replication Key: updated_after, updated_before

Fields

Name	Data Type
id	NUMERIC
body	STRING
author	RECORD
created_at	TIMESTAMP
updated_at	TIMESTAMP
system	BOOLEAN
noteable_id	NUMERIC
noteable_type	STRING
noteable_iid	NUMERIC
resolvable	BOOLEAN
confidential	BOOLEAN
internal	BOOLEAN

Commits

Purpose: This API is used to fetch the commits made in all the projects. In commit responses, created_at and commited_date are identical.

Source API Documentation: https://docs.gitlab.com/ee/api/commits.html

Replication: Key Based Incremental

Replication Key: since, until

Fields

Name	Data Type
id	STRING
short_id	STRING
title	STRING
author_name	STRING
author_email	STRING
authored_date	TIMESTAMP
committer_name	STRING
committer_email	STRING
committed_date	TIMESTAMP
created_at	TIMESTAMP
message	STRING
parent_ids	STRING
web_url	STRING
project_id	STRING

Merge Requests

Purpose: This API is used to fetch all the merge requests made by all users

Source API Documentation: https://docs.gitlab.com/ee/api/merge_requests.html

Replication: Key Based Incremental

Replication Key: updated_after, updated_before

Fields

Name	Data Type
id	NUMERIC
iid	NUMERIC
project_id	NUMERIC
title	STRING
description	STRING
state	STRING
merged_by	RECORD
merge_user	RECORD
merged_at	TIMESTAMP
created_at	TIMESTAMP
updated_at	TIMESTAMP
target_branch	STRING
source_branch	STRING
upvotes	NUMERIC
downvotes	NUMERIC
author	RECORD
assignee	RECORD
assignees	RECORD
reviewers	RECORD
source_project_id	NUMERIC
target_project_id	NUMERIC
labels	STRING
draft	BOOLEAN
work_in_progress	BOOLEAN
milestone	RECORD
merge_when_pipeline_succeeds	BOOLEAN
merge_status	STRING
detailed_merge_status	STRING
sha	STRING
user_notes_count	NUMERIC
should_remove_source_branch	BOOLEAN
force_remove_source_branch	BOOLEAN
allow_collaboration	BOOLEAN
allow_maintainer_to_push	BOOLEAN
web_url	STRING
references	RECORD
time_stats	RECORD
squash	BOOLEAN
task_completion_status	RECORD

Branches

Purpose: This API is used to fetch branches under all projects

Source API Documentation: https://docs.gitlab.com/ee/api/branches.html

Replication: Full Replication

Replication Key: NA

Fields

Name	Data Type
name	STRING
merged	BOOLEAN
protected	BOOLEAN
default	BOOLEAN
developers_can_push	BOOLEAN
developers_can_merge	BOOLEAN
can_push	BOOLEAN
web_url	STRING
commit	RECORD
project_id	STRING

Users

Purpose: This API is used to fetch all users

Source API Documentation: https://docs.gitlab.com/ee/api/users.html

Replication: Full Replication

Replication Key: NA

Fields

Name	Data Type
id	NUMERIC
username	STRING
name	STRING
state	STRING
avatar_url	STRING
web_url	STRING