Data Extraction Services

As part of SDP’s extraction service, data from various sources – unstructured data such as documents, emails, news articles, or feeds and structured enterprise data like client APIs or news feeds – are ingested into the platform using data connectors. SDP also has built-in modules to scrape online public sources such as regulatory portals, company pages, and news sites.    Today, the Internet is rife with much data published directly by governments, businesses, re-publishers, and individuals. The challenge is identifying the right source and selecting data that are accurate and free to use. Straive has developed a scientific method to find the right source for any data-related activity through SDP’s source discovery process. The process identifies the precise storage source for data available and eligible for commercial usage. SDP adopts a multi-source approach for dimensional data to drive accuracy and efficiency.
SDP aggregates thousands of sources across domains to identify business data to meet the needs of clients.  It uses custom-made search queries to aggregate the sources and ranks them based on multiple parameters, which include:
⦁ Authenticity of the source: Legitimacy of the sources based on ownership (directly published or re-published)

⦁ Timeliness of data: Freshness of the data that appear on the source

⦁ Crawling Acceptance: Willingness of the source to allow automatic bots for data extraction

⦁ Volume: Coverage of several entities within the source

⦁ Geography: Coverage of entities from multiple geographies and languages

⦁ Data richness: Comprehensiveness/breadth of the data for a single entity within the source

Straive has a proprietary, mature, and industry-standard module for searching the web and collecting data. The module is used independently for web scraping and monitoring projects for various clients.

Straive’s SDP uses a proprietary-built Crawler Engine to crawl data from various websites. It is used across multiple processes to crawl public information such as metadata, contact profiles, documents, subsidiaries, class action lawsuits, management changes, and journal/book articles from digital repositories. We have significant experience managing/maintaining large/complex directories and monitoring executive movement or tracking events. Our real-time monitoring solution supports critical data needs such as monitoring the price information of stocks and commodities, corporate actions, and time-sensitive schedules.

SDP has been deployed at scale for multiple blue-chip clients and has been used to scrape data from over 12 million web pages. It has collected over 50 million data points around companies, people, products, and locations. Straive uses SDP to monitor around 610,000 web sources daily.

Key Differentiators:

⦁ Provides automated website monitoring feature to track and alert on changes to websites

⦁ Integrates with downstream platforms for cleansing, standardization, and disambiguation

⦁ Scrapes content in other languages, including English, German, Spanish, Italian, Russian, Chinese, Japanese, Korean & Portuguese, without any customization

⦁ Intuitively scrapes paginated sources

⦁ Extracts content via RPA scripts where repetitive actions are involved

⦁ Provides for generic scripts-single scripts extracting information from multiple sites where the same entities are involved

SDP’s PDF extraction service has been developed with years of research and development in content extraction. It is a highly reliable service powered by machine learning cognitive models at the back-end to achieve high PDF/image data extraction accuracy. Natural Language Processing (NLP) techniques, coupled with Optical Character Recognition (OCR) engines, are used to process text and autocorrect defects accurately.
The application is used for high-quality text extraction from typeset PDFs or scanned images/PDFs. Text extraction is possible for multiple languages with a content accuracy of up to 98%.

Key differentiators:

⦁ Conversion Feature for PDF to XML, HTML, text and back to PDF

⦁ Extraction feature for

• TOC, form fields, images, paragraphs, words, section titles, and text from titles
• Text with position coordinates within the PDF 
• Tables, RAW text, meta language, etc.

⦁ Integration with Translation service to support multiple languages

⦁ Ability to add, remove or edit watermarks

⦁ Provision for white-labeled solutions

⦁ Utility features, including page split, password removal, adding of page numbers, and page resizing 

⦁ Keyword generation

⦁ Integration with multiple OCR engines

⦁ Conversion of Images to PDF

The platform is designed to suit different domains. It has several in-built features for table data extraction along with column-spanning and row-spanning options and image extraction processing.  SDP can also zone in on specific areas or ‘ignore’ some text zones to remove them from the output. Currently, close to 70M PDF pages are being processed annually through our PDF extraction module.

Other Unstructured Content 

SDP can process other unstructured content such as emails, XML feeds, word documents, audio, video, and images.

Structured Data Streams 

SDP can ingest structured data feeds such as enterprise data for processing or data management/ enrichment use cases. The data can be ingested via APIs, webhooks, or direct database connections. Data in any convenient, structured format can be directly used for data ingestion. The platform has built-in ETL tools to load data from multiple sources, define complex, automated transformations, test the data pipeline, and load data continuously. In addition, a large selection of native data connectors allows for easy, one-click data ingestion.