Case Study
Scrupp
A Scalable B2B Lead Generation & Data Enrichment Platform
Executive Summary / TL;DR
Scrupp is a comprehensive B2B SaaS platform designed to streamline and automate lead generation and data enrichment processes, primarily leveraging LinkedIn and Sales Navigator. It addresses the critical need for sales, marketing, and recruitment teams to efficiently find, verify, and manage contact and company information at scale. The platform comprises a sophisticated web application, a powerful browser extension, and a robust API. Architected using the Symfony PHP framework on the backend and Vue.js on the frontend, Scrupp employs asynchronous processing via Symfony Messenger for handling intensive tasks like scraping, data enrichment, and email finding, ensuring scalability and responsiveness. Key technical achievements include managing large-scale data operations, complex third-party integrations (payment gateways, data sources, AI tools), a resilient browser extension architecture, and a feature-rich, secure multi-tenant system.
Introduction & Problem Statement
In the competitive B2B landscape, accessing accurate and relevant contact and company information is paramount for driving sales, marketing campaigns, and recruitment efforts. Traditional methods are often manual, time-consuming, error-prone, and struggle to scale. Sales and marketing professionals spend significant time searching for leads, verifying contact details, and enriching company profiles instead of focusing on outreach and relationship building. Key challenges include:
- Inefficiency: Manual prospecting on platforms like LinkedIn is slow.
- Data Accuracy: Contact information found online is often outdated or unverified.
- Scalability: Processing large volumes of leads manually is impractical.
- Data Fragmentation: Lead data often resides in disparate systems or spreadsheets.
- Integration Gaps: Difficulty in seamlessly integrating prospect data into existing CRM or sales workflows.
Scrupp was conceived to directly address these challenges by providing an integrated, automated, and scalable platform for B2B data acquisition and enrichment.
Project Objectives
- Develop a Core Web Application: Create a central hub for users to manage accounts, configure scraping tasks, view results, manage teams, and handle billing.
- Build a Seamless Browser Extension: Enable users to capture leads directly from LinkedIn and Sales Navigator with minimal friction.
- Implement Robust Data Scraping & Enrichment: Develop reliable mechanisms to extract public data from specified sources and enrich it with additional firmographic and contact details (e.g., verified emails).
- Ensure Scalability & Performance: Design the architecture to handle a large user base and intensive background processing tasks without compromising performance or user experience.
- Guarantee Data Quality: Implement processes for email verification and data validation.
- Provide Flexible Export & Integration Options: Allow users to export data in common formats (CSV) and offer API/Webhook capabilities for integration with other tools.
- Offer Secure Multi-Tenancy & Team Features: Enable multiple users within an organization to collaborate securely.
- Integrate Secure Payment Processing: Incorporate reliable payment gateways (Stripe, CoinGate) for subscription management and credit purchases.
The Solution: Scrupp Platform Overview
Scrupp is architected as a multi-component system:
Web Application (Symfony + Vue.js)
The main user interface providing dashboards, task management (Aggregator), contact/company lists (Stream), folder organization, team management, billing, API key management, settings, and administrative controls. It serves as the control center for all platform activities.
Browser Extension (JavaScript)
A lightweight extension (manifest.json indicates Chrome compatibility) that interacts with LinkedIn/Sales Navigator pages. It allows users to initiate scraping/enrichment tasks directly from their browser, sending requests to the backend API and displaying status updates.
Backend API (Symfony + FOSRestBundle)
A versioned RESTful API (src/ApiController/V1/) serves as the communication backbone between the browser extension, the frontend web application, and potentially third-party client applications. It handles authentication, data requests, and task initiation.
Asynchronous Task Processing (Symfony Messenger)
Heavy lifting operations (scraping profiles, finding emails, enriching data, generating exports) are offloaded to background workers managed by Symfony Messenger (src/MessageHandler/, src/Processor/). This prevents blocking web requests, improves UI responsiveness, and allows for horizontal scaling of workers.
Database (PostgreSQL with Doctrine ORM)
A relational database stores user data, accounts, tasks, scraped/enriched data (contacts, companies, emails), billing information, team structures, etc. Doctrine ORM with XML mapping (config/doctrine/) is used for object-relational mapping.
Key Features & Functionality
Based on the file structure (src/Controller/Space/, templates/space/, scrupp.com/features) and inferred functionality:
LinkedIn & Sales Navigator Scraping
Core feature allowing users to extract profiles/company data based on search URLs or direct interaction via the extension.
Bulk Processing (Aggregator)
Users can create "Aggregator" tasks to process large lists of LinkedIn/Sales Navigator URLs or CSV uploads for enrichment.
Contact Email Finding & Verification
Integrates email finding logic (Pattern matching combined with internal verification services) to locate and validate business emails.
Company Data Enrichment
Augments company profiles with details like size, industry, location, funding, technologies used.
Data Segmentation & Management (Stream, Folder)
Allows users to organize contacts and companies into lists/streams and folders for better management.
CSV Export
Flexible export options allowing users to download their data with customizable columns.
Team Collaboration
Functionality for inviting team members, sharing credits/data, and managing permissions.
API Access
Provides API keys for programmatic access to Scrupp's capabilities.
Webhooks
Allows users to receive real-time notifications about completed tasks in external systems.
Subscription & Credit Management
Handles various subscription plans (LTDs, recurring) and credit-based usage via Stripe and potentially crypto (CoinGate).
Third-Party Logins
Supports authentication via Google, LinkedIn, Facebook.
Admin Panel
Comprehensive backend interface for managing users, orders, tasks, coupons, logs, etc.
SEO & Content Features
Includes functionality for generating articles (AI-assisted), managing keywords, and site maps, indicating a content marketing aspect.
Technical Architecture Deep Dive
This section details the core technical decisions and components, crucial for technical assessment.
Backend Framework: Symfony (PHP)
Rationale:
Chosen for its maturity, robustness, extensive ecosystem (Bundles), adherence to standards (PSR), strong security features, built-in support for dependency injection, and excellent integration with Doctrine ORM and Messenger component. Enables rapid development of complex, maintainable applications.
Implementation:
Leverages core Symfony components for Routing, Controllers, Forms, Security, Templating (Twig), Caching, and Console Commands. The structure follows Symfony best practices (src/, config/, templates/, bin/console).
Frontend Framework: Vue.js
Rationale:
Selected for its progressive nature, component-based architecture, reactivity, performance, and ease of integration into existing server-rendered applications (potentially using Twig initially and enhancing with Vue components - assets/js/components/*.vue). Suitable for building interactive dashboards and UI elements.
Implementation:
Used for dynamic UI components like dashboards, modals, real-time updates (e.g., task progress indicators process-worker.vue), and potentially single-page application sections within the user space. Managed via Webpack (webpack.config.js) for building and optimization. jQuery is also present (assets/lib/jquery*), for legacy parts or specific libraries (like datetimepicker, pagepiling).
Database & ORM: Relational DB (PostgreSQL) + Doctrine ORM
Rationale:
Doctrine provides powerful object-relational mapping, abstracting database interactions and simplifying data persistence. It supports complex relationships, lazy loading, caching, and robust migrations. XML mapping (config/doctrine/) offers a declarative way to define entity mappings separate from the code.
Implementation:
Extensive use of Doctrine Entities (src/Entity/) representing core business objects. Repositories (src/Repository/) encapsulate data access logic. Doctrine Migrations (src/Migrations/) manage schema evolution systematically. Custom DQL functions (src/Doctrine/DQL/) extend query capabilities.
Asynchronous Processing: Symfony Messenger
Rationale:
Essential for offloading time-consuming tasks (scraping, API calls, email processing, exports) from the synchronous request-response cycle. Improves application responsiveness and allows for scaling workers independently of the web servers. Provides features like transports (e.g., Redis, RabbitMQ - though specific transport isn't listed, it's configurable via messenger.yaml), retry strategies, and failure handling.
Implementation:
Defines Message classes (e.g., AggregatorProcessor, FindEmails), Handlers (src/MessageHandler/), and Processors (src/Processor/). Commands (src/Command/Workers/) exist to run workers. Used extensively for Aggregator tasks, email finding (EmailFinderHandler), exporting (ExportHandler), and various observers (src/Command/Observer/).
Browser Extension Architecture
Rationale:
Provides seamless integration with the user's browsing workflow on LinkedIn/Sales Navigator.
Implementation:
Standard extension structure (manifest.json, background.js, popup.html). Background scripts (extension/root/js/background/) manage state and communication with the API. Content scripts (extension/root/js/app/front.js, listener.js) interact with the DOM of target pages. Uses JavaScript (ES6+ compiled via Webpack) and communicates with the Scrupp backend API (extension/root/js/api/app.js).
API Design: RESTful API (FOSRestBundle + JMSSerializer)
Rationale:
Provides a standardized way for the frontend, browser extension, and external clients to interact with the backend. FOSRestBundle simplifies REST API development in Symfony, handling content negotiation, routing, and view layers. JMSSerializer provides flexible object serialization/deserialization (JSON/XML).
Implementation:
Versioned API endpoints (src/ApiController/V1/). Uses FOSRestBundle for routing and response formatting (config/packages/fos_rest.yaml). JMSSerializer handles data transformation (config/packages/jms_serializer.yaml). Authentication uses API keys (src/Entity/Custom/ApiKey.php) and potentially session/token auth for the web app.
Third-Party Integrations
- Payment: Stripe (StripeManager.php), CoinGate (CoinGateManager.php), potentially Fondy (PaymentFondyController.php), WayForPay (wayforpay/form.html.twig). Robust handling of subscriptions, one-time payments, coupons (CouponRepository.php).
- Authentication: Google, Facebook, LinkedIn OAuth (knpu_oauth2_client.yaml, src/Controller/Security/).
- Data Sources/Enrichment: Custom API wrappers for Apollo.io (src/Utils/Api/Apollo/), potentially others implicitly used by ScraperManager or EmailFinderManager. Wrappers for LinkedIn (src/Utils/Api/Linkedin/) demonstrate direct interaction attempts (complex due to platform restrictions).
- AI Tools: Integration with AI for article generation (src/Command/SEO/GenerateArticle.php, src/Manager/ThirdParty/AiManager.php) and potentially other tasks.
- Email Delivery: Abstracted mailer system (src/Mailer/) supporting providers like ElasticEmail, Mailjet, SMTP (config/packages/swiftmailer.yaml).
DevOps & Infrastructure (Inferred & Explicit)
- CI/CD: appveyor.yml suggests AppVeyor is used for continuous integration/deployment.
- Build Process: build.sh script for custom build steps. Webpack (webpack.config.js) for frontend assets. Composer for PHP dependencies.
- Testing: PHPUnit (phpunit.xml.dist, bin/phpunit) for backend unit/integration tests.
- Infrastructure: Deployed on cloud infrastructure (AWS) given the need for scalability (web servers, database, message queue, cache, worker instances).
- Monitoring/Logging: Monolog (config/packages/monolog.yaml) for logging. Potential integration with external monitoring services.
Security Considerations
- Symfony's built-in security features (CSRF protection, firewall).
- Use of OAuth 2.0 for third-party logins.
- Secure API key management.
- Input validation (Symfony Forms, src/Utils/Validator.php).
- Use of Recaptcha (beelab_recaptcha2.yaml, google_recaptcha.yaml).
- HTTPS enforcement (standard practice).
- Regular dependency updates (implied by composer.lock, package-lock.json).
Challenges & Solutions
Developing a platform like Scrupp involves significant technical challenges:
Challenge: Scalable and Reliable Scraping
Web scraping, especially from platforms like LinkedIn, is prone to blocking, layout changes, and requires careful handling of sessions/proxies.
Solution:
Implementation of proxy management (src/Entity/Custom/Proxy.php, GetProxy.php), potential use of headless browsers (though not explicit, common for complex JS sites), robust error handling and retry mechanisms within background jobs (Symfony Messenger), session/cookie management (ConvertCookie.php), rate limiting, and potentially distributing tasks across multiple accounts/IPs. Abstracting scraping logic (ScraperManager.php) allows for easier adaptation to source changes.
Challenge: Asynchronous Task Management Complexity
Coordinating potentially thousands of concurrent background jobs, tracking progress, handling failures, and ensuring data consistency.
Solution:
Leveraging Symfony Messenger's features for queuing, transports, retries, and failure handling. Implementing dedicated progress handlers (ProgressHandler.php, EmailFinderProgressHandler.php) to update the frontend. Using database transactions and careful state management within handlers to maintain consistency. Observers (src/Command/Observer/) react to state changes (e.g., payment completion triggering credit addition).
Challenge: Data Volume and Query Performance
Storing and querying millions of contacts, companies, and associated data points efficiently.
Solution:
Careful database schema design, appropriate indexing (potentially leveraging features like FULLTEXT indexes - MatchAgainstFunction.php, or external search engines like Sphinx - symfony_sphinx.yml), optimizing Doctrine queries (using DQL, native queries where needed), implementing pagination (PagerFantaService.php), and potentially using database read replicas for heavy read loads.
Challenge: Email Verification Accuracy
Ensuring a high deliverability rate for found emails.
Solution:
Multi-faceted approach: combining pattern generation, domain checks (MX records), potentially integrating with third-party verification APIs, and implementing feedback loops (tracking bounces reported via webhooks or email replies).
Challenge: Browser Extension Stability & Maintenance
Extensions are sensitive to browser updates and changes on target websites (LinkedIn).
Solution:
Modular JavaScript code (extension/root/js/), careful DOM interaction resilient to minor layout changes, robust error reporting from the extension back to the server, and a streamlined process for updating and deploying new extension versions. Communication relies on the stable backend API.
Challenge: Third-Party API Integration
Managing rate limits, errors, and changes in external APIs (Payment gateways, data providers).
Solution:
Building resilient API clients/wrappers (src/Utils/Api/) with built-in error handling, retries (with exponential backoff), logging, and potentially circuit breaker patterns. Abstracting integrations via Managers (StripeManager, Apollo\ApiWrapper) makes swapping or updating providers easier.
Technology Stack Summary
Results & Impact
Scrupp provides tangible benefits to its users:
- Increased Efficiency: Drastically reduces the time spent on manual prospecting and data entry.
- Improved Data Quality: Delivers verified email addresses and enriched company profiles, leading to better outreach success rates.
- Enhanced Scalability: Enables sales and marketing teams to handle significantly larger lead volumes.
- Streamlined Workflows: Facilitates better organization and management of prospect data, with options for export and integration.
- Facilitated Collaboration: Team features allow organizations to centralize their lead generation efforts.
- Platform Robustness: The architecture demonstrates the ability to build a complex, scalable SaaS application handling intensive background tasks and numerous integrations.
(Quantifiable metrics like user growth, number of contacts processed, average time saved per user would further strengthen this section if available.)
Learnings & Future Directions
The development of Scrupp showcases best practices in building modern, scalable web applications:
- Importance of Asynchronicity: Critical for performance and user experience in data-intensive applications.
- Modular Architecture: Using frameworks like Symfony and Vue.js, along with service-oriented design (src/Manager/), facilitates maintainability and scalability.
- Challenges of External Dependencies: Scraping and third-party API integrations require ongoing maintenance and robust error handling.
- Value of DevOps: CI/CD, automated testing, and infrastructure management are essential for reliable delivery.
Potential Future Directions:
- Deeper CRM integrations (e.g., Salesforce, HubSpot via API - src/Utils/Api/Hubspot/ApiWrapper.php exists).
- More sophisticated AI/ML features (e.g., lead scoring, predictive analytics).
- Expanded data sources beyond LinkedIn.
- Advanced analytics and reporting dashboards.
- WebSocket integration for real-time updates instead of polling.
- Migration to newer framework versions or technologies as appropriate.
Key Areas of Development Focus
Based on the structure, the development heavily focused on:
- Backend Architecture & Business Logic: Implementing the core Symfony application, Doctrine entities/repositories, and service managers.
- Asynchronous System Design: Architecting and implementing the Symfony Messenger setup for background tasks.
- API Development: Creating and maintaining the RESTful API for frontend and extension communication.
- Browser Extension Development: Building the logic for interacting with LinkedIn/Sales Nav and the backend.
- Third-Party Integrations: Implementing reliable wrappers and logic for payment gateways, OAuth providers, and data APIs.
- Database Management: Designing the schema and managing migrations.
- Frontend Development: Creating interactive user interfaces with Vue.js.
This detailed case study provides a comprehensive overview of Scrupp, balancing high-level functional descriptions with in-depth technical analysis, suitable for diverse recruitment audiences. The structure allows for easy parsing by AI to generate visual representations later.